LeTra: a leaf tracking workfow based on convolutional neural networks and intersection over union

Abstract

Background: The study of plant photosynthesis is essential for productivity and yield. Thanks to the development of high‑throughput phenotyping (HTP) facilities, based on chlorophyll fuorescence imaging, photosynthetic traits can be measured in a reliable, reproducible and efcient manner. In most state‑of‑the‑art HTP platforms, these traits are automatedly analyzed at individual plant level, but information at leaf level is often restricted by the use of manual annotation. Automated leaf tracking over time is therefore highly desired. Methods for tracking individual leaves are still uncommon, convoluted, or require large datasets. Hence, applications and libraries with diferent techniques are required. New phenotyping platforms are initiated now more frequently than ever; however, the application of advanced computer vision techniques, such as convolutional neural networks, is still growing at a slow pace. Here, we provide a method for leaf segmentation and tracking through the fne‑tuning of Mask R‑CNN and intersection over union as a solution for leaf tracking on top‑down images of plants. We also provide datasets and code for train‑ing and testing on both detection and tracking of individual leaves, aiming to stimulate the community to expand the current methodologies on this topic. Results: We tested the results for detection and segmentation on 523 Arabidopsis thaliana leaves at three diferent stages of development from which we obtained a mean F‑score of 0.956 on detection and 0.844 on segmentation overlap through the intersection over union (IoU). On the tracking side, we tested nine diferent plants with 191 leaves. A total of 161 leaves were tracked without issues, accounting to a total of 84.29% correct tracking, and a Higher Order Tracking Accuracy (HOTA) of 0.846. In our case study, leaf age and leaf order infuenced photosynthetic capacity and photosynthetic response to light treatments. Leaf‑dependent photosynthesis varies according to the genetic background. Conclusion: The method provided is robust for leaf tracking on top‑down images. Although one of the strong components of the method is the low requirement in training data to achieve a good base result (based on fne‑tuning), most of the tracking issues found could be solved by expanding the training dataset for the Mask R‑CNN model. Keywords: Phenotyping, Tracking, Photosynthesis, Convolutional neural networks, Arabidopsis, Image analysis