Hey guys,
I have been waiting for GSoC to begin in order to present a more concrete proposal.
So, I have discussed my idea with João S. Bueno and he agreed to being my mentor for GSoC. We live nearby and have the same Alma Mater (UNICAMP), so it was easy to get in touch.
Basically, I have proposed to first merge the soc-2009-siox-drb branch with the main branch and then implement my algorithm as another backend for the foreground selection tool. For the first part, it would be nice if I could get some advice from Gerald if possible.
In my proposal I have stated everything that I told you guys on this thread, so there are not many surprises. Also, João and I did some brainstorming and came up with a few ideas which are not described on the attached file, but that I'll try to implement if there's time.
Best regards,
Thiago
Thiago Vallin Spina
MSc student
Laboratory of Visual Informatics
Institute of Computing -- Unicamp
www.liv.ic.unicamp.br
Merging of the detail refinement brush branch and implementation of an IFT-based segmentation tool = Abstract = This proposal aims at first merging the code from the detail refinement brush, of the foreground selection tool, with the main development branch. Then, an algorithm of hard and soft object segmentation based on the Image Foresting Transform will be implemented as an alternative to the current SIOX-based method of extraction. = Background = I was born and raised in Brazil, and I got a Computer Science Bachelor's degree at the University of Campinas (UNICAMP - from Brazil, we have a great history of participations on GSoC). Currently, I have just been admitted to the Computer Science MSc program at UNICAMP, but I've been researching image processing algorithms (mostly related to foreground segmentation on images and videos) for the past 2 years. Before that I worked with robotics, so I have experience on: * Building C/C++ modules applied to indoor navigation of robots for about 2 years as a Research Scholarship Holder -- This project was also applied to the mix of robotics and entertainment, and involved a lot of C/Shell Script hacking and kernel adaptation (also some Java coding). * Developing image and video segmentation algorithms [1,2,3] using the Image Foresting Transform (created by my advisor prof. Alexandre X. Falcão [4]) mostly in C/C++ (some Python) -- Also as a Research Scholarship Holder. In that project I have also gained a lot of experience on code optimization using mostly gcc vector extensions for SIMD. * Dealing with libraries such as wxWidgets (Gtk) to implement the GUI of a software used in my image processing research. = Motivation = Foreground segmentation is an important part of image processing and has been a crucial topic of research for the past decades, but it is still an open problem. The foreground selection tool implemented on GIMP is based on the Simple Interactive Object Extraction (SIOX) algorithm. On the last year's Google Summer of Code, that tool was improved to do some detail refinement on the hard segmentation. That improvement was implemented on a separate git branch named soc-2009-siox-drb, and now there is the need to merge it with the main development branch for the next release of GIMP. Thus, one of the goals of this proposal is to merge those branches. Although SIOX does a great job when segmenting the desired foreground, specially on noisy images, it presents some limitations related to color similarity between foreground and background. In such cases it may take a lot of user interaction to obtain a satisfactory segmentation result. The main goal of my current research is to make user interaction more effective and simple for both hard and soft image/video segmentation. In that spirit, the Image Foresting Transform is a great graph-based alternative to algorithms using the graph cut technique (such as GrabCut [5]), and it was proven to provide better results [6]. IFT is a generalization of the Dijkstra's algorithm, which works quite fast and has served for many purposes in image processing, image analysis, and pattern recognition. The IFT-based framework for hard image segmentation that I have been developing is able to deal with the shortcomings of SIOX, as I have demonstrated on this video: http://vimeo.com/9803589. Thus, the second goal of this proposal is to implement my framework as an alternative to SIOX. Finally, I have talked to João S. Bueno about this proposal, because it is not on the GIMP's ideas list, and he has agreed to being my mentor. Also, he studied at UNICAMP and lives nearby, so communication will be easier. = Implementation and GUI = First, the implementation of the detail refinement brush, developed for the 2009 Google Summer of Code, is going to be merged with GIMP's main development branch. Then, given that the current code for the foreground selection tool is prepared for multiple algorithms, the choice between SIOX and IFT will be done using a combo box or a similar widget. When selected, the IFT-based foreground extraction will work as described on [2]: 1) Once the tool is loaded the image is going to be automatically pre-segmented into a few regions and the result are going to be shown to the user. 2) The user will then draw foreground markers (strokes) selecting the regions which compose the object, and inserting background markers when a region contains both foreground and background pixels (those regions will be split as near to the object's border as possible). 3) Afterwards, the markers are used to improve the automatic pre-segmentation (by learning the color distribution and texture of the object), to select the regions that compose the object, and to divide the ones that need to be divided. 4) Finally, the user may add/remove object and background markers to finalize the hard segmentation (merging/splitting regions). These corrections are local and do not affect other regions. The method has three parameteres which may be selected: * Number of regions which should be pre-segmented (a slider to select between "less" and "more" regions) * The percentage of the object's information that should be used (the color and texture information learned from the object markers is combined with information extracted from the image). * The radius of the strokes. As a post-processing step, the user will be able to draw a new set of strokes which will allow detail refinement on features such as hair, using an IFT-based operator for soft segmentation. There might be a "color" threshold paramater (a percentage actually) for this step, but a default value of 50% yields good results. Furthermore, the IFT-based framework is quite flexible regarding the user interaction style, so the current form used for SIOX of rough foreground selection using a lasso followed by marker drawing can also be used. In fact, this possibility might be evaluated during GSoC. The framework is going to be implemented using GEGL because part of it uses floating point operations, thus, it will be easier to optimize the code using gcc vector extensions. = Timeline = April 26 - May 23: Learn GIMP's API and look at the code from soc-2009-siox-drb branch, analyzing what will be necessary to merge the branches. May 24 - June 25: Merge branches. June 25 - July 10: Start coding the IFT-based framework inside GIMP. July 05 - July 12: Write and submit the mid-term evaluation. July 12 - August 08: Finish coding the IFT-based framework, focusing on the graphical user interface, testing, and documenting. August 09 - 15: Do some code scrubbing and write final evaluation. August 16: Submit final evaluation. August 30: Submit code samples to Google. = Availability = In Brazil we only have one month of Winter break (northern hemisphere Summer) on July, but since I am only taking one class this semester I will be able to work full time on this project. Also, my research will be put partially on hold so that I can have even more time for coding. = References = [1] - http://portal.acm.org/citation.cfm?id=1700473 [2] - http://www.springerlink.com/content/7415482725175nm7/ [3] - R. Minetto, J. P. Papa, T. V. Spina, A. X. Falcão, N. J. Leite, and J. Stolfi. Fast and robust object tracking using image foresting transform. In 16th International Workshop on Systems, Signals and Image Processing, Chalkida, Greece, Jun 2009. IEEE. [4] - http://www.ic.unicamp.br/~afalcao/downloads.html [5] - http://research.microsoft.com/en-us/um/cambridge/projects/visionimagevideoediting/segmentation/grabcut.htm [6] - http://portal.acm.org/citation.cfm?id=1574529
_______________________________________________ Gimp-developer mailing list Gimp-developer@xxxxxxxxxxxxxxxxxxxxxx https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer
