1 Colour - Datasets

Colour science datasets for use with Colour or any Python package manipulating colours. The datasets are hosted in Zenodo under the Colour Science - Datasets community.

It is open source and freely available under the New BSD License terms.

1.1 Features

Colour - Datasets was created to overcome issues encountered frequently when trying to access or use colour science datasets:

  • No straightforward ingestion path for dataset content.

  • No simple loading mechanism for dataset content.

  • Unavailability of the dataset, e.g. download url is down, dataset content is passed directly from hand to hand.

  • No information regarding the definitive origination of the dataset.

Colour - Datasets offers all the above: it allows users to ingest and load colour science datasets with a single function call. The datasets information is hosted on Zenodo where the record for a dataset typically contain:

  • An urls.txt file describing the urls to source the dataset files from.

  • A copy of those files in the eventuality where the source files are not available or the content has changed without notice.

  • Information about the authors, content and licensing.

When no explicit licensing information is available, the dataset adopts the Other (Not Open) licensing scheme, implying that assessing usage conditions is at the sole discretion of the users.

1.1.1 Examples

Colour - Datasets can also be used online with Google Colab.

Most of the objects are available from the colour_datasets namespace:

>>> import colour_datasets

The available datasets are listed with the colour_datasets.datasets() definition:

>>> print(colour_datasets.datasets())

colour-science-datasets
=======================

Datasets : 21
Synced   : 1
URL      : https://zenodo.org/communities/colour-science-datasets/

Datasets
--------

[ ] 3269926 : Agfa IT8.7/2 Set - Marszalec (n.d.)
[ ] 3245883 : Camera Spectral Sensitivity Database - Jiang et al. (2013)
[ ] 3367463 : Constant Hue Loci Data - Hung and Berns (1995)
[ ] 3362536 : Constant Perceived-Hue Data - Ebner and Fairchild (1998)
[ ] 3270903 : Corresponding-Colour Datasets - Luo and Rhodes (1999)
[ ] 3269920 : Forest Colors - Jaaskelainen et al. (1994)
[ ] 4394536 : LUTCHI Colour Appearance Data - Luo and Rhodes (1997)
[x] 3245875 : Labsphere SRS-99-020 - Labsphere (2019)
[ ] 3269924 : Lumber Spectra - Hiltunen (n.d.)
[ ] 4051012 : Measured Commercial LED Spectra - Brendel (2020)
[ ] 3269918 : Munsell Colors Glossy (All) (Spectrofotometer Measured) - Orava (n.d.)
[ ] 3269916 : Munsell Colors Glossy (Spectrofotometer Measured) - Haanpalo (n.d.)
[ ] 3269914 : Munsell Colors Matt (AOTF Measured) - Hauta-Kasari (n.d.)
[ ] 3269912 : Munsell Colors Matt (Spectrofotometer Measured) - Hauta-Kasari (n.d.)
[ ] 3245895 : New Color Specifications for ColorChecker SG and Classic Charts - X-Rite (2016)
[ ] 3252742 : Observer Function Database - Asano (2015)
[ ] 3269922 : Paper Spectra - Haanpalo (n.d.)
[ ] 3372171 : RAW to ACES Utility Data - Dyer et al. (2017)
[ ] 4642271 : Spectral Database of Commonly Used Cine Lighting - Karge et al. (2015)
[ ] 4297288 : Spectral Sensitivity Database - Zhao et al. (2009)
[ ] 4050598 : Spectral Upsampling Coefficient Tables - Jakob and Hanika. (2019)

A ticked checkbox means that the particular dataset has been synced locally. A dataset is loaded by using its unique number: 3245895:

>>> print(colour_datasets.load('3245895').keys())

Pulling "New Color Specifications for ColorChecker SG and Classic Charts" record content...
Downloading "urls.txt" file: 8.19kB [00:01, 5.05kB/s]
Downloading "ColorChecker24_After_Nov2014.zip" file: 8.19kB [00:01, 6.52kB/s]
Downloading "ColorChecker24_Before_Nov2014.zip" file: 8.19kB [00:01, 7.66kB/s]
Downloading "ColorCheckerSG_After_Nov2014.zip" file: 8.19kB [00:01, 7.62kB/s]
Downloading "ColorCheckerSG_Before_Nov2014.zip" file: 8.19kB [00:00, 9.39kB/s]
Unpacking "/Users/kelsolaar/.colour-science/colour-datasets/3245895/dataset/ColorCheckerSG_Before_Nov2014.zip" archive...
Unpacking "/Users/kelsolaar/.colour-science/colour-datasets/3245895/dataset/ColorCheckerSG_After_Nov2014.zip" archive...
Unpacking "/Users/kelsolaar/.colour-science/colour-datasets/3245895/dataset/ColorChecker24_After_Nov2014.zip" archive...
Unpacking "/Users/kelsolaar/.colour-science/colour-datasets/3245895/dataset/ColorChecker24_Before_Nov2014.zip" archive...
odict_keys(['ColorChecker24 - After November 2014', 'ColorChecker24 - Before November 2014', 'ColorCheckerSG - After November 2014', 'ColorCheckerSG - Before November 2014'])

Alternatively, a dataset can be loaded by using its full title: New Color Specifications for ColorChecker SG and Classic Charts - X-Rite (2016)

>>> print(colour_datasets.load('3245895').keys())
odict_keys(['ColorChecker24 - After November 2014', 'ColorChecker24 - Before November 2014', 'ColorCheckerSG - After November 2014', 'ColorCheckerSG - Before November 2014'])

1.2 User Guide

User Guide

The user guide provides an overview of Colour - Datasets and explains important concepts and features, details can be found in the API Reference.

Installation Guide

Primary Dependencies

Colour - Datasets requires various dependencies in order to run:

Pypi

Once the dependencies are satisfied, Colour - Datasets can be installed from the Python Package Index by issuing this command in a shell:

pip install --user colour-datasets

The overall development dependencies are installed as follows:

pip install --user 'colour-datasets[development]'

Bibliography

Asa15

Yuta Asano. Individual Colorimetric Observers for Personalized Color Imaging. PhD thesis, R.I.T., 2015.

Bre20

Harald Brendel. Measured commercial led spectra. April 2020. URL: https://haraldbrendel.com/files/led_spd_350_700.csv (visited on 2020-09-26).

Bre87

Edwin J. Breneman. Corresponding chromaticities for different states of adaptation to complex visual fields. Journal of the Optical Society of America A, 4(6):1115, June 1987. doi:10.1364/JOSAA.4.001115.

DFI+17

Scott Dyer, Alexander Forsythe, Jonathon Irons, Thomas Mansencal, and Miaoqi Zhu. Raw to aces utility data. 2017.

EF98

Fritz Ebner and Mark D. Fairchild. Finding constant hue surfaces in color space. In Giordano B. Beretta and Reiner Eschbach, editors, Proc. SPIE 3300, Color Imaging: Device-Independent Color, Color Hardcopy, and Graphic Arts III, (2 January 1998), 107–117. January 1998. doi:10.1117/12.298269.

HUniversityoKuopioa

Jouni Haanpalo and University of Kuopio. Munsell colors glossy (spectrofotometer measured). doi:10.5281/zenodo.3269916.

HUniversityoKuopiob

Jouni Haanpalo and University of Kuopio. Paper spectra. doi:10.5281/zenodo.3269922.

HUniversityoKuopioc

Jouni Hiltunen and University of Kuopio. Lumber spectra. doi:10.5281/zenodo.3269924.

HB95

Po-Chieh Hung and Roy S. Berns. Determination of constant hue loci for a crt gamut and their predictions using color appearance spaces. Color Research & Application, 20(5):285–295, October 1995. doi:10.1002/col.5080200506.

JH19

Wenzel Jakob and Johannes Hanika. A low-dimensional function space for efficient spectral upsampling. Computer Graphics Forum, 38(2):147–155, May 2019. doi:10.1111/cgf.13626.

JLGS13

Jun Jiang, Dengyu Liu, Jinwei Gu, and Sabine Susstrunk. What is the space of spectral sensitivity functions for digital color cameras? In 2013 IEEE Workshop on Applications of Computer Vision (WACV), 168–179. IEEE, January 2013. doi:10.1109/WACV.2013.6475015.

LR97

M Ronnier Luo and Peter A. Rhodes. Using the lutchi colour appearance data. 1997. URL: https://web.archive.org/web/20040212195937/http://colour.derby.ac.uk:80/colour/info/lutchi/ (visited on 2019-09-10).

LCR+91a

M. Ronnier Luo, Anthony A. Clarke, Peter A. Rhodes, André Schappo, Stephen A. R. Scrivener, and Chris J. Tait. Quantifying colour appearance. part i. lutchi colour appearance data. Color Research & Application, 16(3):166–180, June 1991. doi:10.1002/col.5080160307.

LCR+91b

M. Ronnier Luo, Anthony A. Clarke, Peter A. Rhodes, André Schappo, Stephen A.R. Scrivener, and Chris J. Tait. Quantifying colour appearance. part ii. testing colour models performance using lutchi colour appearance data. Color Research & Application, 16(3):181–197, 1991. doi:10.1002/col.5080160308.

LGR+93

M. Ronnier Luo, X. Wang Gao, Peter A. Rhodes, H. John Xin, Anthony A. Clarke, and Stephen A.R. Scrivener. Quantifying colour appearance. part iii. supplementary lutchi colour appearance data. Color Research & Application, 18(2):98–113, 1993. doi:10.1002/col.5080180207.

LR99

M. Ronnier Luo and Peter A. Rhodes. Corresponding-colour datasets. Color Research & Application, 24(4):295–296, August 1999. doi:10.1002/(SICI)1520-6378(199908)24:4<295::AID-COL10>3.0.CO;2-K.

MUniversityoKuopio

Elzbieta Marszalec and University of Kuopio. Agfa it8.7/2 set. doi:10.5281/zenodo.3269926.

MMT76

John J. McCann, Suzanne P. McKee, and Thomas H Taylor. Quantitative studies in retinex theory a comparison between theoretical predictions and observer responses to the "color mondrian" experiments. Vision Research, 16(5):445–IN3, January 1976. doi:10.1016/0042-6989(76)90020-1.

OUniversityoKuopio

Joni Orava and University of Kuopio. Munsell colors glossy (all) (spectrofotometer measured). doi:10.5281/zenodo.3269918.

SUniversityoKuopio

Raimo Silvennoinen and University of Kuopio. Forest colors. doi:10.5281/zenodo.3269920.

ZKTI09

Hongxun Zhao, Rei Kawakami, Robby T Tan, and Katsushi Ikeuchi. Estimating basis functions for spectral sensitivity of digital cameras. 2009.

HautaKasariUniversityoKuopioa

Markku Hauta-Kasari and University of Kuopio. Munsell colors matt (aotf measured). doi:10.5281/zenodo.3269914.

HautaKasariUniversityoKuopiob

Markku Hauta-Kasari and University of Kuopio. Munsell colors matt (spectrofotometer measured). doi:10.5281/zenodo.3269912.

Labsphere19

Labsphere. Labsphere srs-99-020. 2019. doi:10.5281/zenodo.3245875.

OpenpyxlDevelopers19

Openpyxl Developers. Openpyxl. 2019. URL: https://bitbucket.org/openpyxl/openpyxl/.

XRite16

X-Rite. New color specifications for colorchecker sg and classic charts. 2016. URL: http://xritephoto.com/ph_product_overview.aspx?ID=938\&Action=Support\&SupportID=5884\# (visited on 2018-10-29).

Indirect References

Some extra references used in the codebase but not directly part of the public api:

1.3 API Reference

API Reference

Colour - Datasets

Datasets & Dataset Loading
Loading a Dataset

colour_datasets

load(dataset)

Load given dataset: The dataset is pulled locally, i.e. synced if required and then its data is loaded.

Ancillary Objects

colour_datasets.loaders

DATASET_LOADERS

Dataset loaders ids and callables.

AbstractDatasetLoader(record)

Define the base class for a dataset loader.
Datasets

colour_datasets.loaders

Spectral Upsampling Coefficient Tables - Jakob and Hanika (2019)

DatasetLoader_Jakob2019()

Define the Jakob and Hanika (2019) Spectral Upsampling Coefficient Tables dataset loader.

build_Jakob2019([load])

Singleton factory that builds the Jakob and Hanika (2019) Spectral Upsampling Coefficient Tables dataset loader.
Agfa IT8.7/2 Set - Marszalec (n.d.)

DatasetLoader_AgfaIT872Set()

Defines the University of Kuopio Agfa IT8.7/2 Set dataset loader.

build_AgfaIT872Set([load])

Singleton factory that the builds University of Kuopio Agfa IT8.7/2 Set dataset loader.
Camera Spectral Sensitivity Database - Jiang et al. (2013)

DatasetLoader_Jiang2013()

Define the Jiang et al. (2013) Camera Spectral Sensitivity Database dataset loader.

build_Jiang2013([load])

Singleton factory that builds the Jiang et al. (2013) Camera Spectral Sensitivity Database dataset loader.
Constant Hue Loci Data - Hung and Berns (1995)

DatasetLoader_Hung1995()

Define the Hung and Berns (1995) Constant Hue Loci Data dataset loader.

build_Hung1995([load])

Singleton factory that builds the Hung and Berns (1995) Constant Hue Loci Data dataset loader.
Constant Perceived-Hue Data - Ebner and Fairchild (1998)

DatasetLoader_Ebner1998()

Define the Ebner and Fairchild (1998) Constant Perceived-Hue Data dataset loader.

build_Ebner1998([load])

Singleton factory that builds the Ebner and Fairchild (1998) Constant Perceived-Hue Data dataset loader.
Corresponding-Colour Datasets - Luo and Rhodes (1999)

DatasetLoader_Luo1999()

Define the Luo and Rhodes (1999) Corresponding-Colour Datasets dataset loader.

build_Luo1999([load])

Singleton factory that the builds Luo and Rhodes (1999) Corresponding-Colour Datasets dataset loader.
Forest Colors - Jaaskelainen et al. (1994)

DatasetLoader_ForestColors()

Defines the University of Kuopio Forest Colors dataset loader.

build_ForestColors([load])

Singleton factory that the builds University of Kuopio Forest Colors dataset loader.
LUTCHI Colour Appearance Data - Luo and Rhodes (1997)

DatasetLoader_Luo1997()

Define the Luo and Rhodes (1997) LUTCHI Colour Appearance Data dataset loader.

build_Luo1997([load])

Singleton factory that the builds Luo and Rhodes (1997) LUTCHI Colour Appearance Data dataset loader.
Labsphere SRS-99-020 - Labsphere (2019)

DatasetLoader_Labsphere2019()

Define the Labsphere (2019) Labsphere SRS-99-020 dataset loader.

build_Labsphere2019([load])

Singleton factory that builds the Labsphere (2019) Labsphere SRS-99-020 dataset loader.
Lumber Spectra - Hiltunen (n.d.)

DatasetLoader_LumberSpectra()

Defines the University of Kuopio Lumber Spectra dataset loader.

build_LumberSpectra([load])

Singleton factory that the builds University of Kuopio Lumber Spectra dataset loader.
Measured Commercial LED Spectra - Brendel (2020)

DatasetLoader_Brendel2020()

Define the Brendel (2020) Measured Commercial LED Spectra dataset loader.

build_Brendel2020([load])

Singleton factory that builds the Brendel (2020) Measured Commercial LED Spectra dataset loader.
Munsell Colors Glossy (All) (Spectrofotometer Measured) - Orava (n.d.)

DatasetLoader_MunsellColorsGlossyAllSpectrofotometerMeasured()

Defines the University of Kuopio Munsell Colors Glossy (All) (Spectrofotometer Measured) dataset loader.

build_MunsellColorsGlossyAllSpectrofotometerMeasured([load])

Singleton factory that the builds University of Kuopio Munsell Colors Glossy (All) (Spectrofotometer Measured) dataset loader.
Munsell Colors Glossy (Spectrofotometer Measured) - Haanpalo (n.d.)

DatasetLoader_MunsellColorsGlossySpectrofotometerMeasured()

Defines the University of Kuopio Munsell Colors Glossy (Spectrofotometer Measured) dataset loader.

build_MunsellColorsGlossySpectrofotometerMeasured([load])

Singleton factory that the builds University of Kuopio Munsell Colors Glossy (Spectrofotometer Measured) dataset loader.
Munsell Colors Matt (AOTF Measured) - Hauta-Kasari (n.d.)

DatasetLoader_MunsellColorsMattAOTFMeasured()

Defines the University of Kuopio Munsell Colors Matt (AOTF Measured) dataset loader.

build_MunsellColorsMattAOTFMeasured([load])

Singleton factory that the builds University of Kuopio Munsell Colors Matt (AOTF Measured) dataset loader.
Munsell Colors Matt (Spectrofotometer Measured) - Hauta-Kasari (n.d.)

DatasetLoader_MunsellColorsMattSpectrofotometerMeasured()

Defines the University of Kuopio Munsell Colors Matt (Spectrofotometer Measured) dataset loader.

build_MunsellColorsMattSpectrofotometerMeasured([load])

Singleton factory that the builds University of Kuopio Munsell Colors Matt (Spectrofotometer Measured) dataset loader.
New Color Specifications for ColorChecker SG and Classic Charts - X-Rite (2016)

DatasetLoader_XRite2016()

Define the X-Rite (2016) New Color Specifications for ColorChecker SG and Classic Charts dataset loader.

build_XRite2016([load])

Singleton factory that the builds X-Rite (2016) New Color Specifications for ColorChecker SG and Classic Charts dataset loader.
Observer Function Database - Asano (2015)

DatasetLoader_Asano2015()

Define the Asano (2015) Observer Function Database dataset loader.

build_Asano2015([load])

Singleton factory that the builds Asano (2015) Observer Function Database dataset loader.
Paper Spectra - Haanpalo (n.d.)

DatasetLoader_PaperSpectra()

Defines the University of Kuopio Paper Spectra dataset loader.

build_PaperSpectra([load])

Singleton factory that the builds University of Kuopio Paper Spectra dataset loader.
RAW to ACES Utility Data - Dyer et al. (2017)

DatasetLoader_Dyer2017()

Define the Dyer et al. (2017) RAW to ACES Utility Data dataset loader.

build_Dyer2017([load])

Singleton factory that builds the Dyer et al. (2017) RAW to ACES Utility Data dataset loader.
Spectral Database of Commonly Used Cine Lighting - Karge et al. (2015)

DatasetLoader_Zhao2009()

Define the Zhao et al. (2009) Spectral Sensitivity Database dataset loader.

build_Zhao2009([load])

Singleton factory that builds the Zhao et al. (2009) Spectral Sensitivity Database dataset loader.
Spectral Sensitivity Database - Zhao et al. (2009)

DatasetLoader_Zhao2009()

Define the Zhao et al. (2009) Spectral Sensitivity Database dataset loader.

build_Zhao2009([load])

Singleton factory that builds the Zhao et al. (2009) Spectral Sensitivity Database dataset loader.
Zenodo Records & Community
Configuration

colour_datasets

Configuration([configuration])

Colour - Datasets configuration factory based on colour.utilities.Structure class and allowing to access key values using dot syntax.

sandbox([api_url, community])

A context manager and decorator temporarily setting the configuration to the Zenodo sandbox.

colour_datasets.records

use_sandbox([state, api_url, community])

Modify the Colour - Datasets configuration to use Zenodo sandbox.
Record

colour_datasets

Record(data[, configuration])

Define an object storing a Zenodo record data and providing methods to sync it in a local repository.
Community

colour_datasets

Community(data[, configuration])

Define an object storing a Zenodo community data.

datasets()

Singleton factory that returns Zenodo community that holds the datasets information.
Utilities
Common

colour_datasets.utilities

json_open(url[, retries])

Open given url and return its content as JSON.

hash_md5(filename[, chunk_size])

Compute the Message Digest 5 (MD5) hash of given file.

suppress_stdout()

A context manager and decorator temporarily suppressing standard output.

url_download(url, filename[, md5, retries])

Download given url and saves its content at given file.
Spreadsheet

colour_datasets.utilities

row_to_index(row)

Return the 0-based index of given row name.

index_to_row(index)

Return the row name of given 0-based index.

column_to_index(column)

Return the 0-based index of given column letters.

index_to_column(index)

Return the column letters of given 0-based index.

cell_range_values(sheet, cell_range)

Return given workbook sheet cell range values, i.e. the values of the rows and columns for given cell range.

Indices and tables

1.4 Code of Conduct

The Code of Conduct, adapted from the Contributor Covenant 1.4, is available on the Code of Conduct page.

1.5 Contact & Social

The Colour Developers can be reached via different means:

1.6 About

Colour - Datasets by Colour Developers
Copyright 2019 Colour Developers – colour-developers@colour-science.org
This software is released under terms of New BSD License: https://opensource.org/licenses/BSD-3-Clause
https://github.com/colour-science/colour-datasets