Release April 15, 2008

MATLAB Toolbox for the LabelMe Image Database
written by Antonio Torralba, Bryan Russell and William Freeman
(c) 2008 MIT, Computer Science and Artificial Intelligence Laboratory.


LabelMe is a WEB-based image annotation tool that allows researchers to label images and share the annotations with the rest of the community. If you use this toolbox, we only ask you to contribute to the database, from time to time, by using the labeling tool.

Citation

If you use this dataset or the functions on this toolbox, we would appreciate if you cite:

B. C. Russell, A. Torralba, K. P. Murphy, W. T. Freeman, LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, pages 157-173, Volume 77, Numbers 1-3, May, 2008. (PDF)

Contribute to the dataset

If you find this dataset useful, you can help us to make it larger by visiting the annotation tool and labeling several objects. Even if your contribution seems small compared to the size of the dataset, everything counts! We also welcome submissions of copyright free images. Your annotations and images will be made available for download inmediately.

Here there are some examples of annotated images:

Content

1. Download
2. A quick look into the dataset
3. List of functions in the toolbox
4. Using the toolbox

4.1. Basic functions: indexing, queries
4.2. Searching objects by viewpoint
4.3. Depth ordering of polygons
4.4. Object part hierarchies
5. Communcation with the online tool


Download the LabelMe toolbox

Download the toolbox and add it to the Matlab path.


A quick look into the dataset

The toolbox allows you using the dataset online without needing to download it first. Just execute the next lines to visualize the content of one of the folders of the collection:

HOMEANNOTATIONS = 'http://labelme.csail.mit.edu/Annotations';
HOMEIMAGES = 'http://labelme.csail.mit.edu/Images';
D = LMdatabase(HOMEANNOTATIONS, {'static_street_statacenter_cambridge_outdoor_2005'});
LMdbshowscenes(D, HOMEIMAGES);

(Installing a local copy of the database will allow you a faster access and will reduce the load on our server).


3. List of functions in the toolbox

Introduction
demo - contains examples of how to use several functions. To see more examples visit our FAQ page.


LabelMe tools for reading and ploting the annotations of individual files
LMread - reads one image and the corresponding annotation XML file
LMplot - image and polygons visualization

LabelMe Database tools
LMdatabase - loads all the annotations into a big database struct
LMdbshowscenes - shows thumbnails for the all the images in the database
LMdbshowobjects - shows crops of the objects in the database.
LMobjectnames - returns a list with the name of all the objects in the database
LMobjectindex - retuns the indices of an object class within the annotation struct
LMcountobject - counts the number of instances of an object class in every image

Search tools
LMquery - performs a query on the database using any field

LMobjectpolygon - search for objects using shape matching
qLabelme - search using the online search tool (it does not use the local index)

Image manipulation
LMimread - reads one image from the database
LMimscale - scales the image and the corresponding annotation
LMimcrop - crops the image and the corresponding annotation
LMimpad - pads an image with PADVAL and modifies the annotation

Annotation consistency and synonym lists
LMreplaceobjectname - replaces object names
LMaddtags - Replaces LabelMe object descriptions with the names in the list tags.txt. You can extend this list to include more synonyms. Use this function to reduce the variability on object labels used to describe the same object class. However, the original labelme descriptions contain information that is more specific and you might want to generate other tag files to account for a specific level of description.

Objects, polygons and segmentation
LMobjectpolygon - returns all the polygons for an object class within an image
LMobjectboundingbox - returns bounding boxes

LMobjectmask - returns the segmentation mask for all object instances of one class within an image
LMobjectsinsideimage - removes polygons outside the image boundary

LMobjectcrop - crops one selected object
  
LM2segments - transforms all the labelme labels into segmentation masks (it takes into account occlusions).
  
LMsortlayers - returns the same annotation file, but the objects are sorted by depth.  

Creation of training and test images and databases
LMcookimage - reformat an image and annotation to fit certain requirements.
LMcookdatabase - create tuned databases (you can control the difficulty, the object size, ...)

Communication with online annotation tool
LMphotoalbum - creates a web page with thumbnails and connected to LabelMe online.
LMthumbnailsbar - creates a bar of thumbnails connected to LabelMe online
urldir - like 'dir' but takes as argument a web address
LMgetfolderlist - returns the list of folders from the online database

Install and update images and annotations
LMinstall - installs the database
LMupdate - authomatic update of the annotations for specific files
LMprogressiveinstall - will update your local copy of the images with only the new images and also download all of the annotations

Translation from/to other formats
PAS2LM - Translates PASCAL format to LabelMe
LM2OpenCV - will output a query in format usable for OpenCV Haar

Evaluation for object detection
LMrecallPrecision - Precision-recall curve

XML tools (translates the XML files into MATLAB struct arrays)
writeXML - translates a struct array into a XML file
loadXML - read XML file
drawXML - shows the image and plots the polygons
xml2struct - translates a XML string into a matlab struct
struct2xml - translates a matlab struct into a XML string

Object detection code
A simple object detector using boosting - object detection with boosting trained with this dataset

4. Using the toolbox

First, you need to download the LabelMe database or you need to provide the url of the images and annotations online. If you are going to use the dataset frequently, it is better that you get a local copy.

Define the root folder for the images

HOMEIMAGES = 'C:\yourpath\Images'; % you can set here your default folder
HOMEANNOTATIONS = 'C:\yourpath\Annotations'; % you can set here your default folder

4.1 - Basic functions: indexing, queries

The most common functions are LMdatabase, LMimread, and LMquery.


% This line reads the entire database into a Matlab struct
Dlabelme = LMdatabase(HOMEANNOTATIONS);

% Street scenes
% To get images that have trees, buildings and roads:
[D,j1] = LMquery(Dlabelme, 'object.name', 'building');
[D,j2] = LMquery(Dlabelme, 'object.name', 'road');
[D,j3] = LMquery(Dlabelme, 'object.name', 'tree');
j = intersect(intersect(j1,j2),j3);
LMdbshowscenes(LMquery(Dlabelme(j), 'object.name', 'car,building,road,tree'), HOMEIMAGES);


% You can query the database looking for specific objects and visualize each instance in isolation:
LMdbshowobjects(LMquery(database, 'object.name', 'bottle'), HOMEIMAGES);


4.2 - Synonyms and unification of the LabelMe descriptions

As there are not specific instructions about how labels should be introduced when using the online annotation tool, this results in different text descriptions used for the same object category. For instace, a person can be described as a "person", "pedestrian", "person walking", "kid", etc.

The function LMaddtags replaces LabelMe object descriptions with the names in the list tags.txt. You can extend this list to include more synonyms. Use this function to reduce the variability on object labels used to describe the same object class. However, the original labelme descriptions contain information that is more specific and you might want to generate other tag files to account for a specific level of description.

4.3 - Searching objects by viewpoint

Some objects have information related to viewpoint in the annotation file. In the current setup, viewpoints are discretized in 12 different orientations. For instance, for cars we have:

In order to get all the frontal cars you can use the query:

D = LMquery(Dlabelme, 'object.name', 'car+az270deg'');

We have a tool that allows labeling viewpoints for new objects. If you want to add viewpoint information for new objects send us an email and we will send you a link to the tool. By using the online tool, the data that you will provide can be used in the future by other researchers.

4.3 Searching fully annotated images

 

4.4 - Depth ordering

Frequently, an image will contain many partially overlapping polygons. This situation arises when users complete an occluded boundary or when labeling large regions containing small occluding objects. In these situations we need to know which polygon is on top in order to assign the image pixels to the correct object label.

The function LMsortlayers will sort the polygons accorting to their relative depth ordering. The function uses some heuristics (see paper) and it will not be correct all the time.

4.5 - Object part hierarchies

Despite that object parts are not explicitly labeled as such by the annotation tool, it is posible to authomatically discover object-part hierarchies. When two polygons have a high degree of overlap across many images, this provides evidence for an object-part hierarchy. The toolbox provides a set of functions to suggest parts for an object category. The function partsof will add a new field to each object.

The online query tool shows object parts whenever they are labeled. Here there are some examples:

In order to add parts to the object 'car', run the next line:

D = partsof(Dlabelme, 'car', 'wheel,door,window,tire,mirror,license+plate,windshield,headlight,light');

5 - Communication with the online tool

You can look in the demo.m file, at the end there is a description that explains how can you talk, using Matlab, with the online annotation tool to label only the images that you want and how to get back just those labels and update you annotation file. The effect is like if the annotation tool was just working for you, but still all the annotations that you will enter will be shared and stored online.

The first function you have to look at is:

LMphotoalbum(folderlist, filelist, webpagename, HOMEIMAGES);

This function will create a web page with thumbnails with the images that you want (look at demo.m to see how to use this with LMquery). If you click in any thumbnail it will open LabelMe online showing that image and allowing you to add polygons to it.

Once you are done labeling images, you can update the annotations using:

LMupdate(folderlist, filelist, HOMEIMAGES, HOMEANNOTATIONS);

This function reads the annotations from the web and replaces your local files with the new ones. So, you can get inmediately all the labels you enter.

The result is that the system will behave as if you had a local copy of the annotation tool. And still, everything that you will label, will be shared.

The next command creates a web page that adds thumbnails to the LabelMe annotation tool:

LMthumbnailsbar(folderlist, filelist, 'myphotoalbum.html', HOMEIMAGES);

The web page created will look like this:


(c) 2008 MIT, Computer Science and Artificial Intelligence Laboratory.