Towards automated annotation of benthic survey images: variability of human experts and operational modes of automation

Beijbom, Oscar, Edmunds, Peter J., Roelfsema, Chris, Smith, Jennifer, Kline, David I., Neal, Benjamin P., Dunlap, Matthew J., Moriarty, Vincent, Fan, Tung-Yung, Tan, Chih-Jui, Chan, Stephen, Treibitz, Tali, Gamst, Anthony, Mitchell, B. Greg and Kriegman, David (2015) Towards automated annotation of benthic survey images: variability of human experts and operational modes of automation. PLoS One, 10 7: e0130312. doi:10.1371/journal.pone.0130312


Author Beijbom, Oscar
Edmunds, Peter J.
Roelfsema, Chris
Smith, Jennifer
Kline, David I.
Neal, Benjamin P.
Dunlap, Matthew J.
Moriarty, Vincent
Fan, Tung-Yung
Tan, Chih-Jui
Chan, Stephen
Treibitz, Tali
Gamst, Anthony
Mitchell, B. Greg
Kriegman, David
Title Towards automated annotation of benthic survey images: variability of human experts and operational modes of automation
Journal name PLoS One   Check publisher's open access policy
ISSN 1932-6203
Publication date 2015-07-01
Sub-type Article (original research)
DOI 10.1371/journal.pone.0130312
Open Access Status DOI
Volume 10
Issue 7
Start page e0130312
Total pages 22
Place of publication San Francisco, CA, United States
Publisher Public Library of Science
Language eng
Subject 1300 Biochemistry, Genetics and Molecular Biology
1100 Agricultural and Biological Sciences
Abstract Global climate change and other anthropogenic stressors have heightened the need to rapidly characterize ecological changes in marine benthic communities across large scales. Digital photography enables rapid collection of survey images to meet this need, but the subsequent image annotation is typically a time consuming, manual task. We investigated the feasibility of using automated point-annotation to expedite cover estimation of the 17 dominant benthic categories from survey-images captured at four Pacific coral reefs. Inter- and intra- annotator variability among six human experts was quantified and compared to semi- and fully- automated annotation methods, which are made available at coralnet.ucsd.edu. Our results indicate high expert agreement for identification of coral genera, but lower agreement for algal functional groups, in particular between turf algae and crustose coralline algae. This indicates the need for unequivocal definitions of algal groups, careful training of multiple annotators, and enhanced imaging technology. Semi-automated annotation, where 50% of the annotation decisions were performed automatically, yielded cover estimate errors comparable to those of the human experts. Furthermore, fully-automated annotation yielded rapid, unbiased cover estimates but with increased variance. These results show that automated annotation can increase spatial coverage and decrease time and financial outlay for image-based reef surveys.
Keyword Coral reefs
Climate change
Classification
Cohen's kappa
Cover
Prevalence
Agreement
Error
Q-Index Code C1
Q-Index Status Confirmed Code
Institutional Status UQ

 
Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 18 times in Thomson Reuters Web of Science Article | Citations
Scopus Citation Count Cited 25 times in Scopus Article | Citations
Google Scholar Search Google Scholar
Created: Sun, 23 Aug 2015, 11:31:31 EST by System User on behalf of Scholarly Communication and Digitisation Service