Search by Sketch
March 19, 2010
SEARCH BY SKETCH R&D TEAM | After several months of intense development alongside Stacey Kuznetsov, the Rapid Content-Based Image Search (RCBIS) engine was ready for use in user oriented programs. The most obvious and appealing project was to develop an image search engine which took not text, but a sketch as input. Search by Sketch was born – developed by Stacey Kuznetsov, Teobaldo Fernandez and myself.
Mirroring the RCBIS color model, a limited array of colors is available to users. A usability study found that users wanted to draw unnecessary detail, which wasted time and actually hurt search accuracy. Only one, large brush size is provided to preclude this behavior. The red ‘X’ icon clears the sketch.
Users can draw into two different sketch panels. The upper panel is used to search for image content. Images that closely matched this sketch are ranked highly in the search results. However, images that have commonalities with the sketch drawn in the lower panel are negatively weighted in the search results. These two sketch panels provide a mechanism that acts like a boolean search.
Search results are presented in order of relevance, with the closest matches at the top. To provide additional detail for probable matches and include as many results on screen as possible, thumbnail size is varied. Since our image search algorithm is supposed to be fast (as per the ‘Rapid’ in ‘RCBIS’), we can hit the DB many times a second. This allows us to present live search results as users are sketching their input.
Note: Some people may look at the search results and think that many (or most) of the images don’t match in the slightest. It is important to keep in mind that beyond the intended image (often the top result), no other images may look similar in the dataset. The test database contained about 3000 images. During the video, the database was queried once per second while a user was sketching their query; the database was remote. The polling frequency can be configured, typically set at less than 200 ms if the DB is local.
Video Source: Chris Harrison