An Indexing and Querying System for Online Images
Based on the PNG Format and Embedded Metadata

Jane Hunter jane@dstc.edu.au
Zhimin Zhan z.zhan@dstc.edu.au

DSTC Pty Ltd, Uni. of Queensland, Qld, 4072, Australia. Phone +617 33654310, Fax +617 33654311

September, 1999

Keywords

PNG, Image, Metadata, Standards, Dublin Core

Abstract

In this paper we describe an indexing, querying and browsing system for online images based on the PNG (Portable Network Graphics) image format and its ability to embed structured metadata within the image.

First we describe a Java application which enables the computer-assisted generation and editing of Dublin Core-based metadata descriptions for digital images and the annotation of regions within images. This application integrates an image display window with a graphical user interface and metadata input forms generated from a hierarchical Resource Description Framework (RDF) schema. The schema definition is also used to validate the input descriptions and control the format of the output. At "Save" time, the image is converted from GIF or JPEG to PNG format and the validated metadata which has been input is embedded within the image.

Secondly we describe an image-capable search engine developed through a simple code extension to DSTC's existing HotMeta web-page search engine. HotMeta crawls across the WWW extracting metatags from web pages and storing them in an indexed repository to enable searching. A simple code extension to HotMeta has enabled image search capabilities. Whenever HotMeta encounters a PNG image, it opens the image, extracts the metadata and saves this in the indexed repository.

Finally we describe an improved image browsing method which exploits the metadata embedded in thumbnail PNG images. Clicking on a PNG thumbnail runs a cgi script which opens the thumbnail image, extracts the metadata and displays the full scale JPEG image, with annotated image maps, and other relevant embedded metadata - eliminating the need for backend databases or static web pages.

This prototype system has been developed using digital images of historical photographs from the State Library of Queensland's (SLQ) John Oxley library. In particular, we have used historical photographs of Queensland from the William Boag Photographic Collection from the 1870's.

Table of Contents

  • 1. Introduction
  • 2. Objectives
  • 3. Image Metadata Standards
  • 3.1 Existing Image Metadata Standards
  • 3.2 Dublin Core Qualifiers for Images
  • 3.3 Metadata Model for Annotated Images
  • 3.4 Example Image Description
  • 4. The Peggie Application
  • 5. The Search Engine
  • 6. The Browse Interface
  • 7. Conclusions and Future Work
  • References

    • 1. Introduction

    Image libraries have been enthusiastically embracing new digital technologies to increase the accessibility of their collections through online exhibitions on the Internet. Most online image collections are based on either static web pages of browsable GIF thumbnails linked to full-size JPEG images or search interfaces which use cgi-scripts to access proprietary back-end databases containing the libraries' catalogues.

    In this paper we describe an indexing and retrieval system for online images based on the ability of the PNG format to embed metadata within the image file.

    PNG [HREF1] is the Portable Network Graphics format, a format for storing digital images. It was designed to be the successor to GIF after Unisys and CompuServe suddenly announced in January 1995 that programs implementing GIF would require royalties because of Unisys's patent on the LZW compression method used in GIF. Apart from its freedom from copyright restrictions, PNG has a number of other advantages over GIF. These include: alpha channels (variable transparency), gamma correction (cross-platform control of image brightness), and two-dimensional interlacing (a method of progressive display). PNG also compresses better than GIF (by around 5-25%) and provides three main image types (truecolor, grayscale and palette-based (8-bit)). However the major advantage of PNG which we aim to exploit in the work described in this paper, is it's ability to embed associated metadata within the image.

    First we describe Peggie, an Image Metadata Generator and Editor application which enables the computer-assisted generation and editing of Dublin Core-based metadata descriptions for both digital images and regions within those images. When the images are saved in the PNG format, the associated metadata is saved within the image file.

    This approach then enables existing Dublin Core-based search engines, such as DSTC's HotMeta, to search for images through a fairly simple extension. Whenever the extended HotMeta search engine encounters a PNG image, it opens the file, retrieves the metadata and stores this plus any additional information within its indexed repository enabling image searches.

    Finally we describe an image browsing method which exploits the metadata embedded in thumbnail PNG images. Clicking on a PNG thumbnail runs a cgi script which opens the thumbnail image, extracts the metadata and displays the full scale JPEG image with annotated image map and other associated metadata - eliminating the need for backend databases or static web pages.

    This prototype system has been developed using digital images of historical photographs from the State Library of Queensland's (SLQ) John Oxley library. In particular, we have used historical photographs of Queensland from the William Boag Photographic Collection from the 1870's [HREF2].

    2. Objectives

    Through this work we hope to demonstrate that embedding standardized metadata within images provides the following advantages over existing online image databases:

    In addition to demonstrating the advantages of the embedded metadata approach to image indexing, as listed above, the project had the following objectives:

    3. Image Metadata Standards

    3.1 Existing Image Metadata Standards

    A number of organizations have developed or are investigating metadata standards specifically for image resources. These include:

    Although we are aware of the problems with applying the Dublin Core element set and their qualifiers to resources such as images which often have multiple digital surrogates, we decided to use Dublin Core for the initial prototype for the following reasons:

    Section 3.2 below examines some of the problems associated with applying the DC element set to image resources and specific Dublin Core element qualifiers necessary to satisfy our image description requirements.

    3.2 Dublin Core Qualifiers for Images

    The CNI/OCLC Image Metadata Workshop [HREF8] highlighted some of the major problems with applying Dublin Core, which was designed for the simple resource discovery of textual documents, to image resources. These include: Rather than attempt to solve these problems here we simply propose specific element qualifiers which "refine" rather than "extend" the base elements, enabling straight-forward dumbing-down and the most effective simple resource discovery for our image application. These qualifiers are described below.

    DC.Date

    Date is particularly problematic for images which exist in multiple formats e.g., the original photograph or physical object and multiple digital surrogates. Does it represent the date on which the photograph was taken, scanned or put online? For our application we chose the following representations:

    Date = the date on which the photograph was taken;
    Date.recordCreated = the date on which the metadata record was created;
    Date.placedOnline = the date on which the image was made accessible over the Internet.

    DC.Creator, DC.Publisher, DC.Contributor

    The DC Agent Working Group [HREF11] is attempting to resolve the confusion over what to put in each of these fields. With respect to images, is the Creator the creator of the photograph or the person who scanned it to create the digital surrogate or the person who put the digital surrogate online? In which DC term should the names of these contributors go? For our application, the Creator is the photographer, 'William Boag' and the Publisher is the 'State Library of Queensland'.

    DC.Type

    This defines the category of the resource. For the sake of interoperability, Type should be selected from a hierarchy of enumerated lists. The most recent report by the DC Type Working Group Type suggests an enumerated list for Type values which includes the image type [HREF12]. We suggest that the 'Image' type be further specified by another hierarchical enumerated list as shown:

    DC.Format

    This represents the data format of the resource and can be used to identify the software and possibly hardware that might be needed to display or operate the resource. For the sake of interoperability, the unqualified Format element should be selected from the IANA list of Internet Media Types [HREF13].

    Format = IMT mime type e.g. image/gif, image/jpg

    For images, the top level description may also have to also provide image-specific information such as file size (Kb), image dimensions/spatial resolution (width and height in pixels), and color information for each version of the image.

    Format.filesize = Kb, Mb
    Format.colorpalette = RGB, CMYK, Grayscale
    Format.colourdepth = 8-bit, 24-bit
    Format.dimensions = 1024x768 (spatial resolution)

    DC.Relation

    DC.Relation can be used to specify the location of different versions of an image resource. For example consider the thumbnail version (myimage.gif) of a full-sized image (myimage.jpg):

    For the full-sized image, myimage.jpg, Relation.hasFormat = "myimage.gif"
    For the thumbnail image, myimage.gif, Relation.isFormatOf = "myimage.jpg"

    The Relation qualifiers isPartOf and hasPart can be used to define image items within a collection or to specify spatial regions within an image. For example:

    For the full-sized image, myimage.jpg, Relation.hasPart = "region1, region2, region3"
    For each region, Relation.isPartOf = "myimage.jpg"
    and Coverage.polygon = "x1,y1,x2,y2,x3,y3,..."

    DC.Coverage

    The recommended and most used qualifiers for Coverage are PeriodName and PlaceName.

    However for images, the Coverage qualifiers, Coverage.rect, Coverage.circle, Coverage.point, Coverage.poly can be used to describe the spatial locations and shapes of regions within an image. Given the outline of a region, annotations or descriptions can then be attached to these regions.

    Coverage.rect="x1, y1, x2, y2"
    Coverage.circle="x, y, radius"
    Coverage.point = "x, y"
    Coverage.poly="x1,y1,x2,y2,x3,y3,x4,y4, ...."

    When displaying the image and its metadata though a Web browser, image maps can be created from the second-level region metadata. For example: 

    <MAP NAME="MyMap">
<AREA SHAPE="polygon" HREF="Region1_metadata.html" COORDS="131,294 395,294 395,330 171,330">
<AREA SHAPE="circle" HREF="Region2_metadata.html" COORDS="234,349 15">
<AREA SHAPE="point" HREF="Region3_metadata.html" COORDS="234,349">
<AREA SHAPE="rect" HREF="Region4_metadata.html" COORDS="234,349 361,366">
</MAP>
<IMG USEMAP="#MyMap" SRC="MyImage.jpg">

    3.3 Metadata Model for Annotated Images

    Based on the analyses and requirements descibed above, the optimum schema for this application consists of:
    1. a top-level description applicable to the image as a whole;
    2. optional, second-level descriptions (annotations) associated with a number of spatial regions defined within the image.

    By top-level metadata, we are referring to the metadata description for the full-sized online master JPEG image. The top level scheme consists of the 15 Dublin Core elements (with certain image-specific qualifiers). The second level region metadata consists of a simple 4 element sub-set of the DC element set. Figure 1 below illustrates the proposed data model for the structured image metadata.

    Figure 1: Metadata Model for Annotated Images

    3.4 Example Image Description

    Consider the image 20248.JPG, taken from the William Boag Photograph Collection in the John Oxley library in Brisbane, Qld. It is a digital surrogate of a photograph of a settler family.

    An image map has also been created which attaches metadata to a particular spatial region of the image. This has been defined as Region 1 and is a rectangular region around the head of the woman on the far left of the photograph. Moving the mouse over this region displays the associated region metadata.

    Top Level Metadata Description for Complete Image

    Title: A selector and his family, probably in the Beenleigh district, 1872
    Creator: William Boag
    Subject: Photograph collection - Queensland
    Description: The difficulties faced by a family in the Queensland bush included poor roads, an unreliable mail service and dense, vine-matted scrub. For many years, a selector's staple diet was salted meat (salt horse) and pumpkins. For several months, a woman and her children might be alone in their stringy-bark hut while her husband went off to split shingles or to earn extra money on a cattle property.
    Date.created: 1872
    Date.recordCreated: 1996
    Date.placedOnline: 1997
    Publisher: State Library of Queensland
    Type: image.photograph
    Format: image/jpg
    Format.fileSize: 50.6Kb
    Format.dimensions: 672 x 512
    Format.colorpalette: grayscale
    Identifier: http://archive.dstc.edu.au/RDU/SLQ/boag/20248.jpg
    Source: BOAG negative no. 906
    Language: en
    Relation.isPartOf: http://www.slq.qld.gov.au/jol/boag.htm
    Relation.hasParts: Region1
    Relation.hasFormat: http://archive.dstc.edu.au/RDU/SLQ/boag/20248.gif
    Coverage: Beenleigh region, Queensland, 1872
    Rights: http://www.slq.qld.gov.au/cright.htm

    Secondary Level Metadata for Region1

    Identifier: Region1
    Title: Annie Dickson
    Description: Wife of James Dickson and mother to their 13 children.
    Coverage.rect: 495,207,546,263
    Relation.isPartOf: http://archive.dstc.edu.au/RDU/SLQ/boag/20248.jpg

    4. The "Peggie" Image Metadata Editor and Generator

    The original idea behind the PNG Image Metadata Editor and Generator was to extend DSTC's Reggie application [HREF14], a metadata generator and editor for textual documents, to images. A key difference between the Reggie application and the "Peggie" application was the need for an integrated image display window. This enables the user to simultaneously view the image and input or edit the metadata descriptions.

    Users can open existing GIF or JPEG images, enter the corresponding metadata and then save the image and metadata together as a PNG file. Alternatively, users can edit existing metadata by opening a previously saved PNG file. When a PNG file is opened, the image is displayed in the image panel and the embedded metadata is displayed in the neighbouring form fields. The input form is generated from an RDF schema which corresponds to the data model in Figure 1. The schema also constrains and validates the user input at Save time. Users can also define spatial regions within images and attach metadata to the specified region.

    Figure 2 illustrates the user interface of the Peggie application.

    Figure 2: Top-level Metadata Input Using the Peggie Application

    5. The Search Engine

    Existing Web-based image search and retrieval systems can be classified into four types:

    1. Site-specific search interfaces which access a single or limited number of proprietary backend databases (sometimes via a Z39.50 interface) which represent the image archives from a small number of related organisations or institutions;
    2. Large search engines such as Altavista, Infoseek and Lycos which index the ALT text inside IMG tags on Web pages;
    3. Specialist image search engines like Scour.Net [HREF15], Image Surfer [HREF16] and ditto.com [HREF17] which have built up a very large image databases from crawling the web looking for image files and extracting keywords from the container web page. With each image they typically store a thumbnail image, the URL of the image file, the type of image (e.g. GIF,JPEG), number of search words matched, keywords used in indexing, URL of the web source, size of the image and the date on which it was last found.
    4. Content-based Image Retrieval Systems such as Excalibur Visual RetrievalWare (EVRW)[HREF18] of Excalibur Technologies, Virage Image Search Engine(VISE) [HREF19] of Virage, Inc., Query By Image Content(QBIC) [HREF20] of IBM, VisualSEEK (VSEK)[HREF21] of Columbia University, and Multimedia Analysis and Retrieval System(MARS)[HREF22] of University of Illinois at Urbana-Champaign. These systems provide users with either query images or tools to generate queries based on features such as color histograms, color composition, texture, shape or structure and retrieval based on similarity matching.

    DSTC's existing HotMeta Search Engine crawls over specified web sites, extracting and indexing metadata from embedded HTML Meta tags and saving it in a metadata repository. A simple extension has been made to HotMeta to enable searching for images. Whenever the search engine encounters a PNG image, it opens it, extracts the metadata (if it exists) and stores it in the indexed metadata repository.

    The advantage of the HotMeta PNG image search engine over the image search engines described above is that it makes detailed metadata, usually only available through site-specific proprietary database access, accessible to wide-scale search engines. This approach can provide much better metadata than is currently available via the ALT tag, to the large search engines.

    An online demonstration of the HotMeta PNG version is available [HREF23]. Figures 3 and 4 below illustrate the Search Interface and Query Results from a simple search for the string 'selector'.

    Figure 3: HotMeta Search Interface

    Figure 4: Results of Search on string 'selector'

    6. The Browse Interface

    A browse interface was built which consists of a single web page containing the complete collection as PNG thumbnail images with embedded metadata [HREF24]. Simply clicking on a thumbnail PNG image, runs a cgi script which opens the image, extracts the metadata and dynamically generates a web page displaying the full-sized JPEG image with image maps (if specified) and the associated metadata information.

    The advantages of this approach are two-fold:

    1. If modifications need to be made to the web pages, then they need only be done once in the cgi-script, not to each and every static web page, corresponding to each image. Dynamically-generated web pages are a much more efficient, scalable approach for large image collections;
    2. There is no need for the cgi-script to understand the protocol, query language or fields in the backend database since the metadata is not stored in a database.

    Figure 5 is a screen dump of the browse interface for the Boag Photographic collection. Figure 6 shows the dynamically-generated full-size image and associated metadata which is displayed when the user clicks on the PNG thumbnail for Plate 103 of the collection. Plate 35 of the collection includes an example of a dynamically-generated image map.

    Figure 5: Browse Interface for the William Boag Photographic Collection

    Figure 6: Dynamically-generated Web Page Corresponding to Plate 103

    7. Conclusions and Future Work

    We have developed an application which can be used by image librarians to quickly and easily create and store standardized embedded metadata descriptions within their image collections to improve their discovery and retrieval over the WWW. These descriptions can be used by Dublin Core-based Internet search engines to increase the discovery of these images or to dynamically generate webpages displaying full-screen master images and their associated detailed metadata descriptions.

    We have demonstrated the following advantages of embedded standardized metadata for the resource discovery of images:

    The major disadvantage of this approach is the difficulty associated with performing a batch modification of the metadata for a large collection of images. For example, if the address of the publisher of a large image collection were to change then it would be difficult to perform a blanket change across all of the images. Contrastingly, if the metadata was stored in a database, then this would be a relatively simple procedure.

    Future Work includes:

    Acknowledgements

    The authors wish to acknowledge the use of material belonging to the John Oxley Library. We also wish to thank the staff from the John Oxley Library, in particular Karen Friedl and Niles Elvery, for their assistance.

    The authors also wish to acknowledge the valuable contributions which discussions with Dan Brickley, Dave Beckett and Carl Lagoze have made to this work.

    The authors also wish to acknowledge that this work was carried out within the Cooperative Research Centre for Research Data Networks established under the Australian Government's Cooperative Research Centre (CRC) Program and acknowledge the support of CITEC and the Distributed Systems Technology CRC under which the work described in this paper is administered.

    References

    HREF1
    http://www.graphicswiz.com/png/png.html, 'Portable Network Graphics'.
    HREF2
    http://www.slq.qld.gov.au/jol/picu/boag.htm, John Oxley Library Boag Photograph Collection.
    HREF3
    http://purl.org/DC/, Dublin Core Metadata Initiative.
    HREF4
    http://www.w3.org/TR/REC-rdf-syntax/, RDF Model and Syntax Specification, W3C Recommendation 22 February 1999.
    HREF5
    http://www.dstc.edu.au/Research/Projects/hotmeta/, HotMeta Overview and Capabilities.
    HREF6
    http://www.oberlin.edu/~art/vra/wc1.html VRA Core Categories Version 2.0
    HREF7
    http://vads.ahds.ac.uk/index.html, The Visual Arts Data Service.
    HREF8
    http://www.dlib.org/dlib/january97/oclc/01weibel.html, Image Description on the Internet, A Summary of the CNI/OCLC Image Metadata Workshop, September 24-25, 1996, Dublin, Ohio
    HREF9
    http://www.niso.org/imagerpt.html, NISO/CLIR/RLG Technical Metadata for Images Workshop Report, April 18-19, 1999, Washington DC
    HREF10
    http://www.digitalimaging.org/i_dig2000.html, JPEG 2000 and the Digital Imaging Group: the picture of compatibility.
    HREF11
    http://www.purl.org/dc/groups/agents.htm, Dublin Core Agents Working Group
    HREF12
    http://www.purl.org/dc/documents/wd-typelist.htm, Dublin Core Type Working Group List of Resource Types 1999-08-05
    HREF13
    http://www.isi.edu/in-notes/iana/assignments/media-types/media-types, IANA Registered Media Types
    HREF14
    http://metadata.net/dstc/, 'Reggie, the DSTC Metadata Editor'.
    HREF15
    http://www.scour.net, Scour.net
    HREF16
    http://isurf.interpix.com, Excalibur Image Surfer
    HREF17
    http://www.ditto.com, ditto.com
    HREF18
    http://www.excalib.com/demos.html, 'Excalibur'
    HREF19
    http://www.virage.com", 'Virage'
    HREF20
    http://wwwqbic.almaden.ibm.com", 'QBIC'
    HREF21
    http://www.ctr.columbia.edu/ jrsmith", 'VisualSEEK'.
    HREF22
    http://quark.ifp.uiuc.edu:8080/index.html, 'MARS'.
    HREF23
    http://archive.dstc.edu.au/RDU/HotMeta/png/index.html, PNG-enabled Version of HotMeta
    HREF24
    http://archive.dstc.edu.au/RDU/SLQ/boag/boagphot1.htm, John Oxley Library William Boag Photograph Collection