Abstract：An investigation of benchmark image collectionshow different from digital librariesPurpose This paper aims to introduce the construction methodsimage organizationcollection use and access of benchmark image collections to the digital libraryDLcommunityIt aims to connect two distinct communitiesthe DL community and image processing researchers so that future image collections could be better constructedorganized and managed for both human and computer useDesignmethodologyapproach Image collections are first identified through an extensive literature review of published journal articles and a web searchThena coding scheme focusing on image collectionscreationorganizationaccess and use is developedNextthree major benchmark image collections are analysed based on the proposed coding schemeFinallythe characteristics of benchmark image collections are summarized and compared to DLsFindings Although most of the image collections in DLs are carefully curated and organized using various metadata schema based on an images external features to facilitate human usethe benchmark image collections created for promoting image processing algorithms are annotated on an images content to the pixel levelwhich makes each image collection a more fine-grainedorganized database appropriate for developing automatic techniques on classification summarizationvisualization and content-based retrievalResearch limitationsimplications This paper overviews image collections by their application fieldsThe three most representative natural image collections in general areas are analysed in detail based on a homemade coding schemewhich could be further extendedAlsodomain-specific image collectionssuch as medical image collections or collections for scientific purposesare not coveredPractical implications This paper helps DLs with image collections to understand how benchmark image collections used by current image processing research are createdorganized and managedIt informs multiple parties pertinent to image collections to collaborate on buildingsustainingenriching and providing access to image collectionsOriginalityvalue This paper is the first attempt to review and summarize benchmark image collections for DL managers and developersThe collection creation process and image organization used in these benchmark image collections open a new perspective to digital librarians for their future DL collection developmentLibrary AutomationThrough pixel-level image annotationsit helps automated image classification and accessRetrieval and IndexingThrough content to pixel-level annotationscontent-based image retrieval can be enhancedThis article refers to the use of pixels to comment on the content of imagesThese image collection is suitable for development automatic technologies for classificationabstractvisualization and content retrievalThis involves mode recognition because it involves identification and classification mode or features from data