Semantic parsing for priming object detection in indoors RGB-D scenes

Abstract

The semantic mapping of the environment requires simultaneous segmentation and categorization of the acquired stream of sensory information. The existing methods typically consider the semantic mapping as the final goal and differ in the number and types of considered semantic categories. We envision semantic understanding of the environment as an on-going process and seek representations which can be refined and adapted depending on the task and robot’s interaction with the environment. In this work we propose a novel and efficient method for semantic parsing, which can be adapted to the task at hand and enables localization of objects of interest in indoor environments. For basic mobility tasks we demonstrate how to obtain initial semantic segmentation of the scene into ground, structure, furniture and props categories which constitute the first level of hierarchy. Then, we propose a simple and efficient method for predicting locations of objects that based on their size afford a manipulation task. In our experiments we use the publicly available NYU V2 dataset and obtain better or comparable results than the state of the art at a fraction of the computational cost. We show the generalization of our approach on two more publicly available datasets.

Keywords

Semantic segmentation scene understanding object detection

1. Introduction

In recent years numerous advances have been made in semantic mapping of environments. It has been recognized that the capability of associating semantic concepts with geometric entities in robot’s surroundings can enhance robot’s autonomy and robustness, facilitate more complex tasks and enable better human–robot interaction. These observations led to a large number of approaches, which proposed varying numbers and types of semantic concepts and means of associating them with different parts of the environments. Examples of these included names of rooms/locations and various object and non-object categories. With the exception of a few approaches, semantic parsing at the basic level was formulated as a classification problem, where simple mapping between the sensory data and semantic concepts has been considered. In the proposed work we argue that semantic concepts are naturally arranged in a hierarchy and hierarchical categorization is often more suitable, both in terms of efficiency and adaptability for specific tasks. While in the context of lifelong learning it is desirable to be able to recognize a large number of object categories, it is not necessary to try to recognize them at all times. For example, for basic mobility tasks it is sufficient to be able to understand the free space, walls and obstacles, but it is not necessary to recognize a variety of objects in indoor environments. On the other hand, in the context of object search, more detailed categorization and localization is needed, if one were to facilitate ‘pick up and place’ tasks. In this work we demonstrate how to obtain initial semantic segmentation of the scene into ground, structure, furniture and props categories, which are commonly encountered in indoor environments. These categories along with the geometric features will constitute the first level of hierarchy. This initial semantic segmentation and confidences about presence of objects, will be followed by computation of additional features and generation of hypotheses for objects that based on their size afford a manipulation task.

1.1. Proposed approach

We formulate the semantic labeling problem in the conditional random field (CRF) framework, where the dependencies between random variables are represented by a graph, induced by both image superpixels and 3D scene structure. The distinguishing features of our approach are as follows. (a) The use of a tree graph structure in the CRF setting which effectively approximates the dependencies and enables exact and efficient inference amenable for real-time implementation. (b) The use of simple and efficient appearance and geometric cues, providing evidence about depth discontinuities. We carry out the semantic parsing experiments on the NYU V2 dataset (Silberman et al., 2012)¹, which contains 464 diverse indoor scenes and 1449 annotated frames, achieving superior or comparable performance at the fraction of computational cost. We also show qualitative and quantitative results on the B3DO and UWobj datasets (Janoch et al., 2011; Lai et al., 2011). (c) We show how our system can be integrated with object detectors, in two ways: by pruning the set of candidate locations for less expensive feature computation, and by using the semantic classes’ probabilities outcome as a part of the feature vector.

In the next section, we provide an overview of the related work. In Sections 3 and 4 we describe the details of our approach. Section 5 describes the experiments on the NYU V2 dataset and compares our approach with the state-of-the-art methods. Finally, in Section 6 we present discussions and conclusions of the presented work and discuss possible future directions.

2. Related work

Previous approaches for semantic segmentation varied in the number and type of semantic concepts, means of associating them with different parts of the environments and computational models considered. Examples of the most commonly encountered semantic categories included the names of rooms or locations and various object and non-object categories. In the following section we mostly focus on the review of more recent methods which exploit both 3D geometric and appearance features computed from RGB-D sensors (e.g. Kinect) or obtained by 3D reconstruction.

In indoor environments several methods have been developed exploiting the RGB-D data. Koppula et al. (2011) highlighted the need for efficiency of the final inference and used up to 17 object classes. They were able to exploit stronger appearance and contextual cues due to the scale and different nature of the environment. Recently several researchers carried out more comprehensive experiments on the larger NYU RGB-D dataset introduced by Silberman and Fergus (2011). Ren et al. (2012) focused on local patch-based kernel features and achieved very good average performance while considering 13 structural and furniture classes and grouping all of the smaller objects into ‘other’ categories. The proposed features are computed over high-quality superpixels obtained with computationally expensive boundary detector (Martin et al., 2004). In addition to inference of semantic labels in the work of Silberman et al. (2012) the authors simultaneously considered the problem of inference of the support relations between different semantic categories. The approach relied on an elaborate pre-processing stage involving hierarchical segmentation stage, reasoning about occlusion boundaries and piecewise planar segmentation. All of these stages required a solution to a separate inference problem, using additional features and stage-specific energy functions. In the final inference problem the feature vectors computed over superpixels were over 1000 dimensions. A similar strategy is followed by Gupta et al. (2013) where the authors used the available depth information to improve the initial segmentation, followed by classification of obtained segments. In order to associate disconnected segments belonging to the same object category, they also proposed a long-range amodal completion to improve the segmentation consistency. The above-mentioned approaches relied on improvements of bottom-up segmentation using the depth data as additional cue, followed by classification of the obtained regions. Couprie et al. (2013) bypass the complex feature computation and segmentation stage and use convolutional networks for semantic segmentation. The final hypotheses are then evaluated on over-segmentation for the indoor scenes labeling task.

In contrast to our work the methods reviewed above consider the semantic segmentation as a final goal and are computationally expensive. The proposed approach infers the coarsest level of the semantic hierarchy which can be further refined based on the task.

Another class of approaches for semantic scene understanding focus on the problem of object detection, with or without the bottom-up segmentation stage. The framework based on sliding window object detectors has been explored by Sun et al. (2013) and Lai et al. (2012) using RGB-D sensors. In their work the objects are viewed in a table-top setting at moderate scale. The authors formulated the object detection problem as an inference on a voxel grid, reconstructed from multiple frames of RGB-D data. The final inference is carried in MRF framework, where the data term accumulates evidence from the sliding-window-based detectors trained on different views of the objects. A variant of the HOG descriptor (Lai et al., 2011) was used for capturing the appearance and shape information of each view of an object and trained using support vector machines (SVMs). The larger extent of the objects in the dataset (Lai et al., 2011) and sufficient number of training examples made the use of HOG based detector feasible. Another related work on unsupervised object discovery (Collet et al., 2011) has shown promising results for closer range and small amount of clutter. The work of Mishra et al. (2012) also considers the problem of detection of generic simple objects in an unsupervised setting, relying on the computation of the boundary using both RGB and depth data, followed by a selection of salient points and boundary completion. This methods is very effective for closer range table-top settings, where both depth discontinuities and support surfaces can be well estimated and the process of detection of image contours is more reliable. Their methods relies on a high-quality contour detector (Martin et al., 2004), which is quite expensive to compute. The generic object hypotheses proposed by this detector were then fed to a specific contour-based classifier using a so-called torque descriptor (Teo et al., 2013). The approaches which focus on object detection and categorization are suitable for close-range operation where the place that contains the objects has already been determined and the robot is only deciding which object to pick up.

Several approaches have been proposed for “objectness” detection using image-only information (Alexe et al., 2010; Carreira and Sminchisescu, 2010; Endres and Hoiem, 2010; Cheng et al., 2014). These approaches again are focused on object-centered images and, with the exception of Cheng et al. (2014), typically take of the order of seconds or even minutes to process every frame. In Section 5.2.2 we include comparisons with the recently proposed generic object detector (Cheng et al., 2014), which has shown better performance than similar image-only systems.

The above-cited works differ significantly in the nature of the datasets used to evaluate approaches to semantic segmentation and object detection. The most important distinguishing characteristic is the scale at which objects appear in images. For the datasets, where the objects appear in a table top setting, the challenges come from proper feature selection and efficient categorization in the presence of clutter. The datasets which contain more open scenes focus mostly on improvements in classification of non-object categories (Silberman et al., 2012; Couprie et al., 2013) and large objects appearing in the dataset (Gupta et al., 2013).

In our work we attempt to bridge methods which rely on semantic segmentation of open scenes with larger depth variation and table-top object detection. Instead of striving to achieve high accuracy of object detection for objects appearing at smaller scales in open scenes, we propose a representation and efficient framework which can be used for priming general object search and semantic understanding of non-object components of the environment.

Another line of work attempts to bring in the semantic information to the existing systems and approaches for simultaneous localization and mapping (SLAM). Häne et al. (2013) have proposed a batch algorithm for jointly segmenting the scene and optimize the 3D reconstruction. Their system is a very good option for post-processing but hardly applicable in online robotic operation. An online SLAM system recognizing objects is proposed by Civera et al. (2011), where in parallel to a monocular extended Kalman filter (EKF) SLAM thread there is a thread comparing visual features with the visual features of objects in the database. The recognition thread is always comparing against all of the specific instances and as such is not scalable in the number of different objects and instances. The approach of Salas-Moreno et al. (2013) solves the SLAM problem aided by the a priori knowledge of some objects. In real-time this SLAM system is able to recognize these objects and optimize their pose and the full map given the object models in the database. Although impressive in the results, this approach does not scale well with the number of different objects in the database and it is not suitable for the exploration of new environments or the discovery of new object instances.

In our approach we also strive to develop a framework which is efficient and amenable for real-time implementation. While efficiency has been considered extensively in the context of vision-based SLAM methods, the current techniques for semantic segmentation are computationally expensive and work predominantly in an off-line setting.

The complex issues of representation of semantic knowledge along with the spatial knowledge and the role of ontologies has been pointed out in the work of Pronobis et al. (2010). The authors focused more on exploiting the interplay between room locations and object identities, without considering more detailed semantic segmentation.

3. Semantic segmentation

We formulate the semantic parsing in the framework of CRFs with a tree graph structure encoding the pairwise relationships. We assume that an image and a 3D point cloud of the scene are available. Our approach starts by over-segmenting the image using the efficient simple linear iterative clustering (SLIC) algorithm (Achanta et al., 2012). Every superpixel in the image is interpreted as a cluster in the 3D point cloud for further computations. The 3D centroid of each cluster is used to compute a minimum spanning tree over Euclidean distances, defining the edges for the graphical model. The data and pairwise terms are determined using simple yet discriminative appearance and geometric cues for the classes that we are interested in. The learning and the final inference process are carried out over the graph in the CRFs framework. In the remainder of this section we detail the components of our approach and explain the intuition behind them.

3.1. Framework: CRFs

CRFs are probabilistic undirected graphical models first developed by Lafferty et al. (2001) for labeling sequence data. CRFs are a case of Markov random fields, and thus satisfy the Markov properties. Instead of relying on Bayes’ rule to estimate the distribution over hidden states x from observations z, CRFs directly model p(x|z), the conditional distribution over the hidden variables given observations. Due to this structure, CRFs can handle arbitrary dependencies between the observations. This makes them substantially more flexible when using attributes that are too complex to model their probability distribution and the assumption of independence, as in a naive Bayes setting, is too strong (Koller and Friedman, 2009).

The nodes in a CRF are denoted x =〈x₁, x₂, ⋯, x_n〉, and the observations are denoted z. In our framework the hidden states correspond to the m possible classes: x_i = {ground, structure, furniture, props}.

A CRF factorizes the conditional distribution into a product of potentials. We consider only the potentials for nodes ϕ(x_i, z) (data term) and edges ψ(x_i, x_j, z) (pairwise term). This choice is commonly referred as pairwise CRFs. The potentials are functions that map variable configurations to non-negative numbers capturing the agreement among the involved variables: the larger a potential value, the more likely the configuration. Using the data and pairwise potentials, the conditional distribution over hidden states is written as

p (x | z) = \frac{1}{Z (z)} \underset{i \in N}{Π} ϕ (x_{i}, z) \underset{i, j \in E}{Π} ψ (x_{i}, x_{j}, z)

(1)

where Z(z) is the normalizing partition function, and $〈 N, E 〉$ are the set of nodes and edges on the graph. The computation of this function can be exponential in the size of x. Hence, exact inference is possible for a limited class of CRF models only, e.g. in tree-structured graphs. Potentials are described by log-linear combinations of feature functions, f and g, i.e. the conditional distribution in Equation (1) can be rewritten as

p (x | z) = \frac{1}{Z (z)} \exp (w_{1} \sum_{i \in N} f (x_{i}, z) + w_{2} \sum_{i, j \in E} g (x_{i}, x_{j}, z))

(2)

where [w₁, w₂] are weights representing the importance of each term. CRFs learn these weights discriminatively by maximizing the conditional likelihood of labeled training data. We will describe every term of Equation (2) in detail in Section 3.3. With this formulation we can obtain either the marginal distribution over the class of each variable x_i by solving Equation (2), or the most likely classification of all the hidden variables x. The latter can be formulated as the maximum a posteriori (MAP) problem, seeking the assignment of x for which p(x|z) is maximal.

3.2. Minimum spanning tree over 3D distances

Instead of computing the graph at the pixel level, we over-segment the image into superpixels. The CRF graph structure is typically determined by the neighborhood relations between superpixels. This often yields dense graph structure, see Figure 1(a), which is prone to over connect unrelated classes. We define the graph structure for the CRF as a minimum spanning tree over the Euclidean distances between 3D superpixel’s centroids in a scene. By definition, the minimum spanning tree connects points that are close in the measurement space, highlighting intrinsic locality in the scene, see Figure 1(b). This graph structure was already used in the context of object recognition (Quattoni et al., 2007) and place recognition (Cadena et al., 2012). Cadena et al. (2012) used it for graph matching for place recognition, where the nodes correspond to a sparse set of detected keypoints and the idea was to keep the structure of the graph between the matched nodes in a pair of scenes, while in the current work we handle a denser and structured 3D point cloud and we exploit that physical entities are continuous by nature.

Given that our graph structure is a tree we can use the belief propagation (sum-product) algorithm to exactly infer the marginal distribution of each node, and the max-product algorithm to find the MAP assignment (Koller and Friedman, 2009).

3.3. Feature description

With the graph structure defined for our CRF model, we have to define feature functions f(x, z) and g(x, z) in Equation (2). The features for the data-term are computed as

f (x_{s}, z) = - \log P_{s} (x_{s} | z)

(3)

where the local prior P_s(x_s|z) is the output of a k-nearest neighbors (k-NN) classifier from a set of observations z. We compute P_s(x_s|z) as proposed by Tighe and Lazebnik (2010) in the following equation, where k is fixed to 10:

P_{s} (x_{s} = l_{j} | z) = \frac{1}{\sum_{j = 1}^{m} (\frac{f (l_{j})}{\bar{f} (l_{j})} \frac{\bar{F} (l_{j})}{F (l_{j})})} \frac{f (l_{j})}{\bar{f} (l_{j})} \frac{\bar{F} (l_{j})}{F (l_{j})}

(4)

where f(l_j) (respectively $\bar{f} (l_{j})$ ) is the number of neighbors to s with label l_j (respectively not l_j) in the kd-tree and F(l_j) (respectively $\bar{F} (l_{j})$ ) is the count of all of the observations in the training data with label l_j (respectively not l_j). The observations z computed for every superpixel s capturing the appearance cues obtained from RGB image (image features) and the depth cues (3D features) are summarized in Table 1 and described next.

Table 1.

Local observations.

Default	Observation	Dimensions	Comments
Image features
	LAB color	6	Mean and standard deviation
	y_l	1	Vertical pixel location
	H_s	1	Entropy of hg_s(y)
3D features
	(h_s, d_s)	2	Height and depth
0	$(μ_{Δ d_{s}}, σ_{Δ d_{s}})$	2	if $d_{s} < \frac{1}{‖ N ‖} \sum_{j \in N} (d_{j})$
			Δd_s = ∥d_s − d_j∈N∥
0	$\frac{3 σ_{3}}{σ_{1} + σ_{2} + σ_{3}}$	1	Superpixel planarity
0	1 −mean(∥n_sn_N∥)	1	Neighboring planarity
0	∥n_sj∥	1	Superpixel orientation

3.3.1. Image features

The mean and standard deviation of each channel in the LAB color space for the superpixel.

The vertical pixel coordinate for the superpixel’s centroid.

The entropy of the probability distribution for the superpixel boundaries belonging to the dominant vanishing directions in the scene, see Figure 2 rightmost.

The entropy feature expresses geometric consistency of a superpixel boundary with a particular vanishing direction. This feature is motivated by the observation that in indoor environments superpixel boundaries are often aligned with the vanishing directions. This observation has been also widely utilized in single-view reconstruction techniques (e.g. Hoiem et al., 2007). We employ a five-component gradient mixture model for the Manhattan world described in Coughlan and Yuille (2003). For each image pixel, the model provides the probability of the pixel lying on an edge, probability to pointing to each of the three vanishing points, see Figure 2, and the probability of being noise. We take into account only those pixels having the probability of being on an edge higher than being noise. For each of those points a maximum over last four probabilities is chosen as a membership of the point to either being consistent with one of the three vanishing points or not to be consistent with any, see the fourth column in Figure 2. For a particular superpixel s, we compute a normalized histogram hg_s(y) with four bins y = {1,2,3,4} from memberships of all pixels lying along the superpixel boundary. In order to differentiate between clutter or small objects and structural classes we use the entropy of this normalized histogram,

H_{s} = - \sum_{j = 1}^{4} {hg}_{s} (y = j) \log ({hg}_{s} (y = j))

(5)

3.3.2. 3D features

For the 3D point cloud computed with the depth information we use cues from the 3D position and planarity, for the superpixel itself and for the superpixel with respect to its neighborhood. The cues are as follows.

The depth (d_s) and height (h_s) for the superpixel’s centroid.

The mean and standard deviation of the absolute difference between the depth d_s and the neighborhood’s depths: ||d_s − d_j∈N||. These are only computed if $d_{s} < \frac{1}{‖ N ‖} \sum_{j \in N} (d_{j})$ , with this condition we encode the in-front-of property.

The superpixel planarity encoded by the curvature of a superpixels’ point cloud (Klasing, 2010) using the singular value decomposition (SVD) and sort the singular values such that σ₁ > σ₂ > σ₃.

The neighborhood planarity computed as one minus the mean of the dot product between the normal to the plane against to the neighborhood normals (Zhang et al., 2010).

The superpixel orientation, taken as the projection of the superpixel’s normal on the horizontal plane (Xiao and Quan, 2009).

The superpixel neighborhood N refers to all of the superpixels in contact with superpixel s in the image. In Table 1 we also show the default values and the dimensionality of these observations. As a result we compute for each superpixel 15-dimensional feature vector with 8 elements from image features and 7 from 3D features.

Pairwise term

The pairwise feature g(x_i, x_j, z) is computed for every edge in the graph as

g (x_{i}, x_{j}, z) = {\begin{matrix} 1 - \exp (- ‖ c_{i} - c_{j} ‖_{2}) & \to & l_{i} = l_{j} \\ \exp (- ‖ c_{i} - c_{j} ‖_{2}) & \to & l_{i} \neq l_{j} \end{matrix}

(6)

where ||c_i − c_j||₂ is the L2-norm of the difference between the mean colors of two superpixels in the LAB color space and l is the class label.

Given the graph structure and the features outlined above, we proceed with the description of the learning and inference stage and the performance evaluation of the final semantic segmentation approach in Section 5. An example of the probability distribution outcome from our approach is shown in Figure 3.

4. Priming object detection

In this section we show how the marginals from the semantic segmentation approach can be used in the task of localizing objects in the scene. We propose a simple detector for generic objects and show how to combine it with the semantic segmentation results to improve the detection rate and computational cost. Instead of training an object detector separately for each object category we consider a single category of generic objects whose extent can be well approximated by a bounding box. Each bounding box is then characterized by a boundary feature characterizing the orientation of the boundary close to the bounding box border. The feature is motivated by previous works on saliency and detection of generic objects (Alexe et al., 2010) and works for objects of relatively simple shape, where majority of the true object boundary is close to the bounding box boundary. In the next section we describe in more detail the boundary feature, the generic object detector and the method for proposing and scoring of candidate bounding boxes to generate object hypotheses.

4.1. Features and classifier

We compute the gradient orientations in four blocks depicted in Figure 4(right), obtained by shrinking and enlarging the bounding box by five pixels each, to capture the variation around the actual boundary. In each block we quantize the orientations from 0 to 360° into nine orientation bins. This is done both for intensity and depth channel yielding a 2 × 4 × 9 = 72-dimensional feature.

At this point we can introduce the first modification to include the semantic segmentation information into the feature vector. We compute the probability distribution of the bounding box over the four classes (ground, structure, furniture, props), by averaging and re-normalizing the pixel’s probabilities inside the window. Now, the feature vector is augmented with the four values resulting in a 76-dimensional feature vector.

We train an SVM classifier with histogram intersection kernel (Maji et al., 2008) using the mentioned feature vector. The positive examples come from the ground truth bounding boxes and the negative ones are randomly picked from the generated windows using the proposal of Van de Sande et al. (2011). In the window generation method of Van de Sande et al. (2011) the image is first over-segmented into superpixels, which are then merged using the criteria of similarity over the color space and superpixels’ sizes building a fine-to-coarse hierarchical over-segmentation. The bounding boxes of each superpixel at different levels in the hierarchy are possible candidates.

The ground truth is constrained to those objects with sides smaller that 50 cm and greater than 5 cm. We obtain the metric dimensions of the bounding boxes directly from the depth map. The negative examples are constrained by the same rule.

4.2. Object search strategy

At the test time, instead of generating candidate windows with different predefined aspect ratios and scales, we again use the strategy of Van de Sande et al. (2011) with the same constraints on the size as before.

Now, we introduce the second modification which is sampling the candidate windows with the probability of the bounding box belonging to furniture or props. The idea behind this sampling strategy is that if our semantic segmentation is confident that one candidate box covers the wall or the floor we do not need to evaluate the classifier on it or compute the features for that window. On the other hand, it is highly probable the objects we are looking for lie over furniture pieces or belong to the props semantic class.

5. Experiments

In our experiments we use the NYU V2 RGB-D dataset (Silberman et al., 2012), which contains 1449 labeled frames. The labeling spans over 894 different classes produced using Amazon Mechanical Turk. The authors of the dataset also provide a train and test splits and a mapping from the 894 categories to 4 classes: ground, structure, furniture and props, as was used by Silberman et al. (2012). We take 795 training frames for building the kd-tree and for CRF parameter learning, and the remaining 654 frames for testing and quantitative comparison against state-of-the-art methods in RGB-D semantic segmentation.²

We obtain superpixel segmentation using the SLIC implementation from the VLFeat library of Vedaldi and Fulkerson (2008), followed by the computation of the features described in Table 1. With the computed features in the training set we build a kd-tree using the implementation of Buchanan and Fitzgibbon (2006) with the default parameters. Then for every superpixel in the training set we obtain the k-NN classification for the training data using Equation (4) with the k = 10 nearest neighbors.

The minimum-weight spanning tree (MST) is computed from 3D centroids of all of the superpixels. Now, using the MST graph, the output of the k-NN classifier in Equation (3) and the pairwise potentials (Equation (6)) we learn the parameters in the CRF setting. For the learning, inference and decoding with CRFs we use the Matlab code for undirected graphical models (UGM).³

5.1. Results on semantic segmentation

At testing time, to obtain the most likely label assignment for the superpixels we solve the MAP problem over the CRFs. This problem does not require any threshold selection and all of the parameters are computed/learned from the data. The inference results give us the labeling assignments over superpixels, we transfer those to every pixel in the superpixel to compute the pixel-wise accuracy of semantic labeling. In Figure 5 we show several examples of the output of our approach.

Table 2 shows the confusion matrix normalized by rows (recall on the diagonal). The props class is the most frequently confused. Props class is defined by Silberman et al. (2012) as “small objects that can be easily carried”. We can observe in the results (Figure 5) several planar objects (posters, carpets, placemats) are labeled as structure or furniture. Another source of confusion is due to the quality of the ground truth, where we found several ambiguities in the structure and furniture categories, where kitchen tables, stoves, dishwashers and cabinets were labeled as the two interchangeably.

Table 2.

Confusion matrix for the pixel-wise accuracy in %.

Recall	ground	furniture	props	structure
ground	88.5	8.8	1.3	1.4
furniture	6.6	63.5	12.5	17.4
props	8.1	32.9	31.8	27.2
structure	1.0	14.0	6.3	78.7

In first row of Table 3 we show the pixel-wise recall accuracy along with the average and global accuracy for our approach: CRF-MST and k-NN + SLIC with Image and 3D features. To study the importance of Image features versus 3D features, we remove one set at a time keeping the rest of the system intact. The rows Image features and 3D features show the corresponding performance when using only one type of features. Using only 3D features we can see better results for three out of four classes at the cost of very poor accuracy in props and smaller average and global accuracy. Our full system, with both sets of features obtains the best trade off for all performance measures. In Table 3 we also show the result by only using the data term, k-NN row, using the image and 3D information. It is clear that the MST and the CRF framework improve the general performance.

Table 3.

Pixel-wise percentage recall accuracy.

	ground	furniture	props	structure	Average	Global
Ours	88.5	63.5	31.8	78.7	65.6	67.3
only Image features	68.5	46.1	30.6	75.3	55.1	57.9
only 3D Features	88.9	65.4	18.2	80.3	63.2	66.4
with D_s & no H_s	90.3	64.3	25.3	78.0	64.5	66.6
with D_s	89.4	61.9	30.6	78.3	65.1	66.5
Ours & N-Im graph	88.9	64.9	22.4	81.4	64.4	67.3
Ours & Delaunay graph	89.1	63.6	19.6	81.4	63.5	66.5
k-NN, Equation (3)	88.5	50.2	43.7	72.7	63.8	62.6
Silberman et al. (2012)	68	70	42	59	59.6	58.6
Couprie et al. (2013)	87.3	45.3	35.5	86.1	63.5	64.5

Manhattan world assumption: We implicitly use the Manhattan World assumption when using our entropy feature H_s, Equation (5). This feature helps the system to identify clutter or small objects that are not aligned with the dominant directions in a scene. Here H_s is an image-only feature, but it is also possible to compute dominant plane normals from depth data aligned with the Manhattan World (Silberman et al., 2012). Following Taylor and Cowley (2012) we find the dominant orthogonal directions of the world frame then we normalize the absolute values of the superpixel’s normals projected onto this world frame to compute their entropy, similarly as in the case of H_s:

D_{s} = - \sum_{j = 1}^{3} m_{sj} \log (m_{sj}), with m_{sj} = \frac{| n_{sj} |}{\sum_{k = 1}^{3} | n_{s, k} |}

(7)

We have evaluated our system removing H_s from our features and adding D_s instead, and also using both, i.e. our features and D_s, (with D_s & no H_s) and (with D_s) in Table 3, respectively. In the first case the accuracy for ground is improved but for props is hurt. The second case, the performance for props improves again but in general is not as good as our initial proposal.

Graph structure: In order to compare the advantage of the MST as a graph structure we have also included the results of our system with other two graphs, first the classical choice of connecting each superpixel with its neighbors in the image (N-Im), see Figure 1(a) and the second, by connecting the nodes through the Delaunay triangulation over their 3D positions. Given that both graphs contain loops we have used loopy belief propagation (LBP) for inference. In Table 3 we can see that their average and global accuracies are still competitive but at the expense of poor accuracy in props. They also incur computationally the most expensive inference and no warranty of convergence. For N-Im graph LBP took, with a maximum of 100 iterations, in average 0.96 s per frame and did not converge on 19 out of 654 frames. If we reduce the maximum iterations to 20 we reduce the cost to 0.83 s per frame but in 455 out of 654 frames the inference does not converge. For Delaunay graph LBP took, with a maximum of 100 iterations, on average 2.8 s per frame and did not converge on 15 out of 654 frames. Our MST graph structure allows exact inference using belief propagation in only 42 ms in Matlab.

Comparisons with the state of the art: We also compare against the full method proposed by Couprie et al. (2013). That method uses the four channels (RGB and depth) in a multi-scale convolutional network to learn the features, and a two-layer multi-perceptron as a classifier, followed by the aggregation of the final assignments over superpixels computed by Felzenszwalb and Huttenlocher (2004). We can observe that our approach is competitive or better for all of the classes, with better average and global accuracy. Their system takes 2 days to train but is very efficient in the testing stage, spending 0.7 seconds per frame to perform the segmentation using a parallel implementation for the convolution stage (Farabet et al., 2013).

In comparison with the original work of authors who released this dataset (Silberman et al., 2012), as shown in Table 3, their system still obtains the best accuracy in the furniture and props classes. They use a feature vector of 1128 elements to compute the data term with a logistic regression classifier. The cues come from color (36), shape (1086) and scene (6) information (see the supplementary material in Silberman et al. (2012)). Our representation and features are notably simpler enabling more efficient feature computation and inference while still achieving comparable performance.

5.1.1. Independent datasets

We evaluate our semantic segmentation on two independent datasets: the Berkeley 3-D Object Dataset (B3DO) (Janoch et al., 2011) and the RGB-D Object Dataset from University of Washington (UWobj) (Lai et al., 2011). B3DO with 849 frames has more variability in the indoors scenes and spans from offices to house rooms. The smoothed depth information is obtained by another in-painting technique, different from that used by NYU V2. UWobj is a dataset that focuses on close-range object instance detection and object categorization and contains mostly table-top scenes from the lab with large extent objects. This dataset is released without in-painting in the depth maps. From both datasets we evaluate our semantic segmentation approach on the scenes that correspond to open space, instead to table-top scenes, as our semantic segmentation focuses on semantic classes relevant during navigation or exploration rather than for manipulation. We consider an open space any scene where the median of the depth map is greater than 2.5 meters leaving us with 182 frames from B3DO and 45 frames from UWobj. Some result are shown in Figure 6, we can see a similar performance that for the NYU V2 dataset, even with the differences in depth pre-processing, type of scenes and poses of the sensor, which show that our proposal is a general and powerful tool for semantic segmentation.

5.1.2. Timing

The experiments were carried out with a research implementation of our approach in Matlab. The computational cost, for NYU V2 in testing, is detailed in Figure 7, excluding the superpixel over-segmentation. The system runs on average at 1 fps in a single-thread of a 3.4 GHz IntelCore i7-2600 CPU M350 and 7.8 GB of RAM. Including the SLIC over-segmentation is still able to run at 0.5 fps. For the whole system, the mean and the maximum computational times are 0.7 and 1.0s, respectively. The average cost to obtain the SLIC superpixels is 611 ms, although a C++ implementation would take half of that time as reported by Achanta et al. (2012). In the feature computation the main bottleneck is the computation of the gradient mixture model (Coughlan and Yuille, 2003) for vanishing directions. The training stage takes less than 2 seconds to build the kd-tree and 180 seconds for the CRF parameter learning. In total including the feature computation, the computational cost for all of the training data is less than 20 minutes.

An initial implementation of semantic segmentation approach in C++ and using the GPU implementation of SLIC (gSLIC) of Ren and Reid (2011) is closer to a real-time operation working at around 5 fps. Table 4 shows the mean timing for both versions, Matlab/mex and C++/GPU. A more efficient system is possible by implementing other stages, e.g. k-NN, on the GPU.

Table 4.

Mean computational timing for the semantic segmentation system in milliseconds in the NYU-V2 test set.

	SLIC	F-lab	F-entropy	F-3D	MST	k-NN	Pairwise	MAP	Total
Matlab/mex	611	95	338	173	4	42	4	42	1309
C++/GPU	10.58	3.50	28.57	19.62	0.34	77.97	0.39	0.99	141.0

5.2. Object localization

We train the object detector as described in Section 4 on the training set of the NYU V2 dataset and evaluate it on the test set from the same dataset and on B3DO and UWobj datasets. Figure 8 shows some example results from testing in all of the datasets.

We use the intersection over union measure greater than 0.5 to accept a detection (Everingham et al., 2010). We also use the concept of ground truth bounding boxes to ignore (BB_GT−ig) as proposed by Dollár et al. (2012). Any proposal matched to a BB_GT−ig does not count as good detection or as a false positive. This solves issues for example when we detect individual instances of objects and the BB_GT−ig is covering a group of objects (book versus books).

5.2.1. Performance evaluation

In Figure 9 we show the performance of our approach for priming object detection in terms of F1-score versus detector SVM score, the SVM score is calibrated to span from 0 to 1. In order to show the contribution of using the semantic segmentation information we present three versions for our system, as described in Sections 4.1 and 4.2.

Ours [I, D], No sampling: This is the basic system, descriptor with intensity and depth channels without using the results from the semantic segmentation in any stage.

Ours [I, D, P], No sampling: Here we augment the descriptor with the probabilities from the semantic segmentation.

Ours [I, D, P], Sampling: Here we sample the bounding boxes proposals by p(furniture) + p(props) and augment the descriptor with the probabilities from the semantic segmentation.

The F₁-score curve shows how we improve the baseline system, first by augmenting the descriptor with the probabilities from the semantic segmentation and then by sampling using the p(furniture) + p(props). When setting a robotic system for real operation, we may determine a threshold for the SVM score, Figure 9 depicts that after choosing a threshold on one of the datasets the rank of the three versions is maintained to new environments. The major benefit in the sampling is reflected in the computational cost given the number of discarded candidates before the feature computation. For example, in Figure 10(left), without sampling, the cost for feature computation and SVM evaluation is 760 and 60 ms, respectively. Instead, with the sampling, Figure 10(right), the cost is 440 and 20ms. In Figure 11 we show the frequency of speed-up for all of the datasets. In the worst case there is almost no gain in efficiency in cluttered scenes, and in other scenes (e.g. corridors) we can get speed-ups of up to a factor of four.

The low F₁-scores for the object detector are due to the high proportion of false alarms. We can see in Figure 8 that we obtain several false alarms, but a lot of them are because the ground truth bounding boxes are incomplete and many objects which we successfully detect are not labeled at all. This problem is easy to find in the research community as the labeling is generated only for the specific objects of interest in each paper. However, in real-life applications a robot can find a possibly unlimited number of different objects and instances.

5.2.2. Comparisons

As a comparison with the state-of-the-art object detectors we include the results from the very recent BInarized Normed Gradients (BING) objectness estimation (Cheng et al., 2014). The source code is provided by the authors with a training on the PASCAL-VOC 2007 dataset. In Figures 12, 13 and 14, we depict the performance of the BING detector and of our approach in terms of detection rate (DR) versus false positives per image (fppi) and precision–recall (PR) curves. The DR-fppi and PR curves show that our object proposal system is better than the only-image state of the art. In the UWobj dataset BING is better than ours in one portion of the curve, if more than 50 fppi are allowed in a real system, and the reason for this is that this dataset is the most structured one among the three datasets, with objects in table-top setting and in the center of the image.

6. Discussion

We have shown a basic implementation with real-time capabilities that effectively uses appearance and 3D cues to generate evidence about the structure of the scene, while achieving better average and global accuracy of semantic labeling compared with the state of the art. Note that the accuracy reported by Couprie et al. (2013) was computed after training the system for 849 different classes and, then, the results were clustered into the four categories, indicating that their approach and features are less robust to the high intra-class variability in the four class problem.

We have shown that our graph structure induced by the MST over metric euclidean distances does not sacrifice the labeling accuracy, and keeps the intra-class components coherently connected. Furthermore, by this selection we gain an exact and efficient inference. The computational complexity for the inference is $O ({nm}^{2})$ , where n is the number of nodes in the graph, and m is the number of classes. In that sense our approach is suitable for segmentation problems with a small number of classes. While the computational cost could violate a real-time constraint for problems with a large number of classes, we believe that a reliable semantic segmentation system should follow a coarse-to-fine strategy, where the labels for specific objects should be sought if necessary, and not classify large numbers of different object classes at once.

An interesting discussion is related to the choice of basic representation and the features. Silberman et al. (2012) use over 1000-dimensional feature vectors, concatenating many engineered features, most of them developed previously for diverse but related tasks. On the other hand, Farabet et al. (2013) propose to avoid the feature engineering and learn the features from the data using convolutional networks. They are still using a 768-dimensional feature vector for RGB channels, and the feature vectors have over 1000 dimensions when including the depth channel (Couprie et al., 2013). We can see that in both approaches, feature engineering vs feature learning, high dimensional feature space is constructed in which the different classes are more easily separable. In this work we propose and demonstrate, that fewer (15-dimensional) but more meaningful features are sufficient to obtain better semantic segmentation in RGB-D scenes. In addition to the choice of features there are at least two more reasons for this improvement. First, selecting a well-performing local classifier to handle high intra-class variability: k-NN in our case versus logistic regression (Silberman et al., 2012), versus a two-layer multi-perceptron (Couprie et al., 2013). Second, defining a more natural connection between neighbors given the type of data: MST over 3D distances in our system, versus connecting all of the neighbors in the image (Silberman et al., 2012), versus no connections at all between superpixels (Couprie et al., 2013). Farabet et al. (2013) show a similar comparison.

We have also shown how to integrate the results of semantic segmentation with object detection, proposing a simple detector of generic objects whose shape can be well approximated by a bounding box. The performance of the object detection in terms of precision recall is not on a par with other object detection results where large objects and more discriminative features are considered (Ren et al., 2012). This is partly due to the lack of correctly labeled objects in the dataset (many small objects are part of the background class) and simplistic boundary feature descriptors. More sophisticated descriptors (e.g. Teo et al., 2013) and more accurate labeling can significantly improve the proposed approach.

We plan to explore strategies for obtaining the observations in a multi-scale way to improve the performance, this is inspired on the boosting achieved by Farabet et al. (2013) when comparing the multi-scale versus one-scale convolutional networks.

We expect a C++/GPU implementation for our object detection to work at more than 10 fps, a similar gain as for our semantic segmentation. Note that the BING detector (Cheng et al., 2014) works with image information only, and it is still possible to get a better performance by adapting it to take into account the depth channel, and maybe the semantic class probabilities, although a reduction in its efficiency is expected.

Footnotes

Funding

This research has been funded by the US Army Research Office (grant number W911NF-1110476) and the Australian Research Council (grant number DP130104413).

Notes

References

Achanta

Shaji

Smith

Lucchi

Fua

Süsstrunk

(2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions onPattern Analysis and Machine Intelligence 34(11): 2274–2282.

Alexe

Deselaers

Ferrari

(2010) What is an object? In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp. 73–80.

Buchanan

Fitzgibbon

(2006) Interactive feature tracking using K-D trees and dynamic programming. In: IEEE conference on computer vision and pattern recognition, volume 1, pp. 626–633.

Cadena

Gálvez-López

Tardós

Neira

(2012) Robust place recognition with stereo sequences. IEEE Transaction on Robotics 28(4): 871–885.

Carreira

Sminchisescu

(2010) Constrained parametric min-cuts for automatic object segmentation. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp. 3241–3248.

Cheng

Zhang

Lin

Torr

(2014) BING: Binarized normed gradients for objectness estimation at 300fps. In: IEEE CVPR.

Civera

Galvez-Lopez

Riazuelo

Tardos

Montiel

JMM

(2011) Towards semantic SLAM using a monocular camera. In: 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 1277–1284.

Collet

Srinivasa

Hebert

(2011) Structure discovery in multi-modal data: a region-based approach. In: Proceedings IEEE international conference on robotics and automation.

Coughlan

Yuille

(2003) Manhattan World: Orientation and outlier detection by Bayesian inference. Neural Computation 15(5): 1063–1088.

10.

Couprie

Farabet

Najman

LeCun

(2013) Indoor semantic segmentation using depth information. CoRR, abs/1301.3572.

11.

Dollár

Wojek

Schiele

Perona

(2012) Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 34: 743–761.

12.

Endres

Hoiem

(2010) Category independent object proposals. In: Computer Vision–ECCV 2010. Berlin: Springer, pp. 575–588.

13.

Everingham

Van Gool

Williams

Winn

Zisserman

(2010) The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision 88(2): 303–338.

14.

Farabet

Couprie

Najman

LeCun

(2013) Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 35: 1915–1929.

15.

Felzenszwalb

Huttenlocher

(2004) Efficient graph-based image segmentation. International Journal of Computer Vision 59: 167–181.

16.

Gupta

Arbeláez

Malik

(2013) Perceptual organization and recognition of indoor scenes from RGB-D images. In: IEEE conference on computer vision and pattern recognition (CVPR).

17.

Häne

Zach

Cohen

Angst

Pollefeys

(2013) Joint 3D scene reconstruction and class segmentation. In: IEEE conference on computer vision and pattern recognition (CVPR).

18.

Hoiem

Efros

Hebert

(2007) Recovering surface layout from an image. International Journal of Computer Vision 75: 151–172.

19.

Janoch

Karayev

Jia

. (2011) A category-level 3-D object dataset: Putting the Kinect to work. In: ICCV workshop on consumer depth cameras for computer vision.

20.

Klasing

(2010) Aspects of 3D Perception, Abstraction, and Interpretation in Autonomous Mobile Robotics. PhD Thesis, Technical Univeristy of Munich, Munich, Germany.

21.

Koller

Friedman

(2009) Probabilistic Graphical Models: Principles and Techniques. Cambridge, MA: MIT Press.

22.

Koppula

Anand

Joachims

Saxena

(2011) Semantic labeling of 3D point clouds for indoor scenes. In: 25th annual conference on neural information processing systems.

23.

Lafferty

McCallum

Pereira

(2001) Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the 18th international conference on machine learning. San Francisco, CA: Morgan Kaufmann, pp. 282–289.

24.

Lai

Ren

Fox

(2012) Detection-based object labeling in 3D scenes. In: IEEE international conference on on robotics and automation.

25.

Lai

Liefeng

Xiaofeng

Fox

(2011) A large-scale hierarchical multi-view rgb-d object dataset. In: 2011 IEEE international conference on robotics and automation (ICRA), pp. 1817–1824.

26.

Maji

Berg

Malik

(2008) Classification using intersection kernel support vector machines is efficient. In: IEEE conference on computer vision and pattern recognition, 2008 (CVPR 2008), pp. 1–8.

27.

Martin

Fowlkes

Malik

(2004) Learning to detect natural image boundaries using local brightness, color and texture cue. IEEE transactions on pattern analysis and machine intelligence 26(5): 530–549.

28.

Mishra

Srivastav

Aloimonos

(2012) Segmenting simple objects using RGB-D. In: Proceedings IEEE international conference on robotics and automation.

29.

Pronobis

Sjöö

Aydemir

Bishop

Jensfelt

(2010) representing spatial knowledge in mobile cognitive systems. In: 11th International conference on intelligent autonomous systems (IAS-11).

30.

Quattoni

Wang

Morency

Collins

Darrell

(2007) Hidden conditional random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(10): 1848–1852.

31.

Ren

Reid

(2011) gSLIC: A Real-time Implementation of SLIC Superpixel Segmentation. Technical Report, University of Oxford, Department of Engineering.

32.

Ren

Fox

(2012) RGB-(D) scene labeling: Features and algorithms. In: IEEE conference on computer vision and pattern recognition (CVPR).

33.

Salas-Moreno

Newcombe

Strasdat

Kelly

Davison

(2013) SLAM++: Simultaneous Localisation and Mapping at the level of objects. In: IEEE conference on computer vision and pattern recognition (CVPR).

34.

Silberman

Fergus

(2011) Indoor scene segmentation using a structured light sensor. In: Proceedings of the international conference on computer vision - workshop on 3D representation and recognition.

35.

Silberman

Hoiem

Kohli

Fergus

(2012) Indoor segmentation and support inference from RGBD images. In: ECCV.

36.

Sun

Fox

(2013) Attribute based object identification. In: IEEE international conference on robotics and automation.

37.

Taylor

Cowley

(2012) Parsing indoor scenes using RGB-D imagery. In: Proceedings of robotics: science and systems, Sydney, Australia.

38.

Teo

Myers

Fermüller

Aloimonos

(2013) Embedding high-level information into low level vision: efficient object search in clutter. In: IEEE international conference on robotics and automation.

39.

Tighe

Lazebnik

(2010) Superparsing: Scalable nonparametric image parsing with superpixels. In: Computer Vision - ECCV 2010. Berlin: Springer.

40.

Van de Sande

Uijlings

Gevers

Smeulders

(2011) Segmentation as selective search for object recognition. In: 2011 IEEE international conference on computer vision (ICCV), pp. 1879–1886.

41.

Vedaldi

Fulkerson

(2008) VLFeat: An open and portable library of computer vision algorithms. Avaiilable at: http://www.vlfeat.org/.

42.

Xiao

Quan

(2009) Multiple view semantic segmentation for street view images. In: 2009 IEEE 12th international conference on computer vision, pp. 686–693.

43.

Zhang

Wang

Yang

(2010) Semantic segmentation of urban scenes using dense depth maps. In: Computer Vision - ECCV 2010. Berlin: Springer.