ACOLITE Docker / parallel processing

[quinten]

You can run ACOLITE in Docker to allow easy cross platform deployment or to run multiple instances in parallel. It is not recommended to run multiple instances of other ACOLITE distributions in parallel.

To start it is assumed you have a working Docker installation. The attached zip file can then be easily used to set up a local copy of the ACOLITE Docker image. This zip file has a Dockerfile which gets the Ubuntu 20.04 image, updates it to the latest version, installs Miniconda and the ACOLITE dependencies from conda-forge. It then clones the ACOLITE GitHub code. Inside the zip file are also two empty directories, "input" and "output" and a default "settings" file. The input and output directories will at runtime be used as mount points for respectively the image to be processed and the output directory. The settings file can also be replaced with other versions at runtime, but note that the output option in your settings file should always start with "output=/output".

Set up the ACOLITE docker image - note this will take some time as software is downloaded and installed:

Code: Select all

unzip acolite-docker.zip
cd acolite-docker
docker build -t acolite .

When the ACOLITE Docker image is complete it can be run specifying the paths to an input file and a target output directory:

Code: Select all

input=/path/to/scene
output=/path/to/output
docker run -d -v $input:/input -v $output:/output acolite

Optionally a settings file can be provided - make sure to have the output option in that settings file starting with /output, e.g. "output=/output/custom-settings":

Code: Select all

input=/path/to/scene
output=/path/to/output
settings=/path/to/settings
docker run -d -v $input:/input -v $output:/output -v $settings:/settings acolite

Several processing instances can be started at the same time, e.g. using a list of files and GNU parallel:

Code: Select all

parallel -j 150% docker run -d -v {}:/input -v $output:/output -v $settings:/settings acolite ::: `cat files.txt`

You can save the Docker image, this step will take a while:

Code: Select all

docker save acolite | gzip > acolite-docker.tar.gz

and transfer it to another machine:

Code: Select all

docker load < acolite-docker.tar.gz

I hope this is useful! Please let me know if you have any comments on this approach!

Quinten

[quinten]

Note that the repository for the Landsat and Sentinel-2 code has been changed to https://github.com/acolite/acolite_ls2 and the git clone command in the Dockerfile may need to be adapted from:

Code: Select all

## clone acolite
RUN git clone https://github.com/acolite/acolite

to

Code: Select all

## clone acolite
RUN git clone https://github.com/acolite/acolite_ls2

The https://github.com/acolite/acolite repository will be used for the upcoming generic version of ACOLITE, and when that version is tested enough the clone command can be kept as it was.

Quinten

[quinten]

An update for running the new generic ACOLITE (at https://github.com/acolite/acolite) in a Docker container. Most steps are the same, but since the new ACOLITE also supports processing of single file bundles (e.g. zipped Planet data, or CHRIS HDF files) I changed the handling of the inputfile. Now the directory containing the bundle(s) you want to process should be mounted, with the bundles listed in the settings file with their paths changed to /input/.

Set up the ACOLITE docker from the file attached here:

Code: Select all

unzip acolite-docker-20210715.zip
cd acolite-docker
docker build -t acolite .

Set up variables for the input directory and output directory, as well as a custom settings file. Run the docker, specifying the input and output directories and the settings file:

Code: Select all

input=/path/to/scene_directory
output=/path/to/output_directory
settings=/path/to/settings_file.txt
docker run -d -v $input:/input -v $output:/output -v $settings:/settings acolite

The custom settings_file.txt should contain at least the inputfile and output directory settings. For this Docker set up, these settings have to start with /input/ and /output/ respectively, but may contain additional directory levels. Additional settings may be provided. For example:

Code: Select all

inputfile=/input/Data/31UES/S2A_MSIL1C_20210329T105631_N0209_R094_T31UES_20210329T130721.SAFE
output=/output/Zeebrugge
limit=51.30,3.10,51.42,3.29
dsf_aot_estimate=fixed

Quinten

[quinten]

Since I removed all the LUTs from the main ACOLITE repository, it is advised to set up the Docker with the LUTs you expect to need. The attached file includes lines to comment/uncomment for retrieval of the different sensor LUTs.

For example, these three lines in the Dockerfile are not commented out, so the Docker will be set up with S2 and L8 LUTs:

Code: Select all

RUN conda run -n acolite python ./acolite/launch_acolite.py --retrieve_luts --sensor S2A_MSI
RUN conda run -n acolite python ./acolite/launch_acolite.py --retrieve_luts --sensor S2B_MSI
RUN conda run -n acolite python ./acolite/launch_acolite.py --retrieve_luts --sensor L8_OLI

The Docker should still support processing of other sensors, but will then need to retrieve the LUTs each run.

Quinten

[quinten]

It is probably best to use --no-cache when building the Docker file, so the latest code updates from https://github.com/acolite/acolite are pulled in.

Set up the ACOLITE docker from the file attached in the post above:

Code: Select all

unzip acolite-docker-20210727.zip
cd acolite-docker
docker build --no-cache -t acolite .

Quinten

[quinten]

I have made two pre-built Docker images available at https://hub.docker.com/r/acolite/acolite that can be pulled in directly. The first image (all_20211122) contains all the LUTs to allow ACOLITE/DSF processing of all supported sensors. The second image (tact_20211122) contains the LUTs for Landsat sensors (L5_TM, L7_ETM, L8_OLI) as well as a compiled version of LibRadtran (v.2.0.2, http://www.libradtran.org/doku.php) so it can be used to run both DSF and TACT for Landsat. Note that these require quite a bit of disk space (10+ and 5+ GB). The all_latest and tact_latest images are currently identical to the 20211122 versions. 20211122 is the date of the GitHub clone of the ACOLITE code. The images should be pulled in automatically when running the named Docker image, but can be explicitly downloaded as well (see further).

Edit: The latest version is now 20211122 for small changes to the handling of the upcoming S2 file format/detector footprints. Changed tags in this post to "all_latest" and "tact_latest".

To run ACOLITE Docker, you need to set input and output variables in your shell environment to pass to the Docker, respectively using full paths to the scene to process and the target directory on your host machine. These paths will be mounted to the input/ and output/ directories inside the Docker. For example setting the variables to local paths:

Code: Select all

input=/Users/quinten/LT05_L1TP_201028_20110930_20200820_02_T1
output=/Users/quinten/Docker/

and running TACT without custom settings (--rm to clean up the container after running, -d for detached run):

Code: Select all

docker run --rm -d -v $input:/input -v $output:/output acolite/acolite:tact_latest

You can also pass a custom settings file, but make sure to include the inputfile and output settings as in the example below. These input/ and output/ refer to the directories inside the Docker where the host directories will be mounted. For example the basic settings.txt file for running TACT:

Code: Select all

inputfile=input/
output=output/
tact_run=True
tact_output_atmosphere=True
tact_output_intermediate=True

You can process for a small ROI using the limit (decimal S,W,N,E coordinates, comma separated), and modify the output directory as long as it starts with output/. This directory in output will then be created:

Code: Select all

inputfile=input/
output=output/Gironde/
limit=45.4,-1.5,45.7,-0.8
tact_run=True
tact_output_atmosphere=True
tact_output_intermediate=True

The full path to the custom settings file has to be provided to the Docker as well:

Code: Select all

input=/Users/quinten/LT05_L1TP_201028_20110930_20200820_02_T1
output=/Users/quinten/Docker/
settings=/Users/quinten/settings.txt

and running TACT with these custom settings:

Code: Select all

docker run --rm -d -v $input:/input -v $output:/output -v $settings:/settings acolite/acolite:tact_latest

You can pull in the images directly using

Code: Select all

docker pull acolite/acolite:all_latest

or

Code: Select all

docker pull acolite/acolite:tact_latest

These images are based on the Dockerfiles attached here and can be regenerated locally using:

Code: Select all

unzip acolite-docker-all-20211121.zip
cd acolite-docker-all
docker build --no-cache -t acolite:all_latest .

or

Code: Select all

unzip acolite-docker-tact-20211121.zip
cd acolite-docker-tact
docker build --no-cache -t acolite:tact_latest .

I hope this is useful for someone!

Quinten

[quinten]

Updated Docker images are now available at https://hub.docker.com/r/acolite/acolite

These correspond to the 20220222.0 release of ACOLITE, and include L9 processing for both OLI (DSF) and TIRS (TACT) for the "tact" tag. L9, SuperDove, Quickbird2 and MERIS LUTs were also added to the "all" tag for DSF processing. Attached to this post are the updated files used to generate the images.

Quinten

[quinten]

Updated Docker images are now available at https://hub.docker.com/r/acolite/acolite

These correspond to a current (2023-10-17) clone of ACOLITE. The new images are based on Ubuntu LTS 22.04 and the "tact" tag now uses libRadtran 2.0.5 instead of 2.0.2. VIIRS LUTs were also added to the "all" tag for DSF processing. Attached to this post are the updated files used to generate the images.

Quinten