Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation
Ver/Abrir
Autor/a
Ferrer-Ferrer, Mar
Gregorio, Eduard
Blok, Pieter M.
Hemming, Jochen
Morros, Josep-Ramon
Rosell-Polo, Joan R.
Vilaplana, Verónica
Ruiz-Hidalgo, Javier
Fecha de publicación
2023-04-21ISSN
0168-1699
Resumen
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing
Tipo de documento
Artículo
Versión del documento
Versión publicada
Lengua
Inglés
Materias (CDU)
633 - Cultivos y producciones
Páginas
13
Publicado por
Elsevier
Publicado en
Computers and Electronics in Agriculture
Citación
Gené-Mola, Jordi, Mar Ferrer-Ferrer, Eduard Gregorio, Pieter M. Blok, Jochen Hemming, Josep-Ramon Morros, Joan R. Rosell-Polo, Verónica Vilaplana, and Javier Ruiz-Hidalgo. 2023. "Looking Behind Occlusions: A Study On Amodal Segmentation For Robust On-Tree Apple Fruit Size Estimation". Computers And Electronics In Agriculture 209: 107854. doi:10.1016/j.compag.2023.107854.
Número del acuerdo de la subvención
MICIU/Programa Estatal de I+D+I orientada a los retos de la sociedad/RTI2018-094222-B-100/ES/Tecnologías de agricultura de precisión para optimizar el manejo de dosel foliar y la protección fitosanitaria sostenible en plantaciones de frutales/PAgFRUIT
MICINN/Programa Estatal para impulsar la investigación científico-técnica y su transferencia/PID2021-126648OB-100/ES/Protección de cultivos de precisión para conseguir objetivos del Pacto Verde Europeo en uso eficiente y reducción de fitosanitarios mediate Agricultura de Precisión/PAgPROTECT
MICINN/Programa Estatal de generación del conocimiento y fortalecimiento científico y tecnológico del sistema I+D+I y Programa Estatal de I+D+I orientada a los retos de la sociedad/PID2020-117142GB-I00/ES/ /DeeLight
FEDER/ / /EU/ /
Program
Ús Eficient de l'Aigua en Agricultura
Este ítem aparece en la(s) siguiente(s) colección(ones)
- ARTICLES CIENTÍFICS [2590]
El ítem tiene asociados los siguientes ficheros de licencia:
Excepto si se señala otra cosa, la licencia del ítem se describe como http://creativecommons.org/licenses/by/4.0/