AIROGS: Artificial Intelligence for Robust Glaucoma Screening Challenge
Authors
Coen de VenteKoenraad A. VermeerNicolas JaccardHe WangHongyi SunFiras KhaderDaniel TruhnTemirgali AimyshevYerkebulan ZhanibekulyTien-Dung LeAdrian GaldranMiguel Ángel González BallesterGustavo CarneiroR. G. DevikaHrishikesh Panikkasseril SethumadhavanDensen PuthusseryHong LiuZekang YangSatoshi KondoEdward WangAshritha DurvasulaJónathan HerasMiguel Ángel ZapataTeresa AraújoGuilherme ArestaHrvoje BogunovićMustafa ArikanYeong Chan LeeHyun Bin ChoYoon Ho ChoiAbdul QayyumImran RazzakBram van GinnekenHans G. LemijClara I. Sánchez
Abstract
The early detection of glaucoma is essential in preventing visual impairment. Artificial intelligence (AI) can be used to analyze color fundus photographs (CFPs) in a cost-effective manner, making glaucoma screening more accessible. While AI models for glaucoma screening from CFPs have shown promising results in laboratory settings, their performance decreases significantly in real-world scenarios due to the presence of out-of-distribution and low-quality images. To address this issue, we propose the Artificial Intelligence for Robust Glaucoma Screening (AIROGS) challenge. This challenge includes a large dataset of around 113,000 images from about 60,000 patients and 500 different screening centers, and encourages the development of algorithms that are robust to ungradable and unexpected input data. We evaluated solutions from 14 teams in this paper and found that the best teams performed similarly to a set of 20 expert ophthalmologists and optometrists. The highest-scoring team achieved an area under the receiver operating characteristic curve of 0.99 (95% CI: 0.98-0.99) for detecting ungradable images on-the-fly. Additionally, many of the algorithms showed robust performance when tested on three other publicly available datasets. These results demonstrate the feasibility of robust AI-enabled glaucoma screening.
Why we worked on this
The interesting problem in AIROGS is not classification accuracy. It is what a model does when handed an image it should refuse to judge. Laboratory performance collapses in the field because real screening produces ungradable and out-of-distribution images, and a system that guesses confidently on those is worse than no system. That question — knowing when not to answer — is the same one that governs every production model we deploy.
Where to start
The AI Modernization Assessment is two weeks and $5,000. Not a code audit. You get a prioritized roadmap, effort and cost ranges, and a written list of the workflows where AI would add risk without adding value.
Book the assessment