Hi,
I am running the codes now. I was just wondering is dsf_min_tile_aot=0.01 (in the log file) the same as aot_560? I see that aot_560 is the same all over the scene.
Regards
Asim
Acolite DSF results over Barnets Sea
Re: Acolite DSF results over Barnets Sea
Hi Asim
Indeed setting dsf_aot_estimate=fixed means that there will be one aot value used for the scene. This value is stored in the NetCDF global attributes, but if you use dsf_write_aot_550=True then a separate layer will be included as well. This additional output is not needed for this purpose (and really useful when processing in fixed mode).
The dsf_min_tile_aot=0.01 setting is the minimum value used for the aot for a given tile during tiled processing (dsf_aot_estimate=tiled).
You can find information on these settings in the manual.
Quinten
Indeed setting dsf_aot_estimate=fixed means that there will be one aot value used for the scene. This value is stored in the NetCDF global attributes, but if you use dsf_write_aot_550=True then a separate layer will be included as well. This additional output is not needed for this purpose (and really useful when processing in fixed mode).
The dsf_min_tile_aot=0.01 setting is the minimum value used for the aot for a given tile during tiled processing (dsf_aot_estimate=tiled).
You can find information on these settings in the manual.
Quinten
Re: Acolite DSF results over Barnets Sea
Hi,
I am getting NaN values (for Rrs) by doubling the aot_550. Please see the settings file. On the first run, Rrs values are not NaN. I will now process aot_550 * 1.5 if it works. Any suggestions, please?
Also, it takes too long to process the scene.
inputfile=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Matchups/scenes/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
output=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Acolite_MSI/Matchups_2021/Acolite/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
##limit=
mid_lat=
mid_lon=
l2w_parameters=Rrs_*, chl_oc2, chl_oc3
l2w_export_geotiff=True
aerosol_correction=dark_spectrum
dsf_aot_estimate=fixed
dsf_fixed_aot=0.16080622 # original dsf_fixed_aot * 2 here
dsf_fixed_lut=ACOLITE-LUT-202102-MOD1 # LUT name ("ac_model") from the first run
dsf_write_aot_550 = True
## Apply the preliminary gains by Pahlevan et al. (2017)
gains=False
## output resolution (S2 only 10, 20, or 60 m)
s2_target_res=60
ancillary_data=True
#glint_correction=False
dsf_residual_glint_correction=True
dsf_residual_glint_correction_method=default
l2w_mask_water_parameters=True
l2w_mask_cirrus=True
l2w_mask_high_toa=True
l2w_mask_smooth=True
mask_smooth_sigma=3
merge_tiles=False
l2w_mask_negative_rhow=True
l2w_mask=True
rgb_rhot=False
rgb_rhos=False
map_l2w=False
map_l2r=False
map_title=False
map_colorbar=False
##map_colorbar_orientation=vertical
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
l2w_export_geotiff=True
I am getting NaN values (for Rrs) by doubling the aot_550. Please see the settings file. On the first run, Rrs values are not NaN. I will now process aot_550 * 1.5 if it works. Any suggestions, please?
Also, it takes too long to process the scene.
inputfile=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Matchups/scenes/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
output=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Acolite_MSI/Matchups_2021/Acolite/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
##limit=
mid_lat=
mid_lon=
l2w_parameters=Rrs_*, chl_oc2, chl_oc3
l2w_export_geotiff=True
aerosol_correction=dark_spectrum
dsf_aot_estimate=fixed
dsf_fixed_aot=0.16080622 # original dsf_fixed_aot * 2 here
dsf_fixed_lut=ACOLITE-LUT-202102-MOD1 # LUT name ("ac_model") from the first run
dsf_write_aot_550 = True
## Apply the preliminary gains by Pahlevan et al. (2017)
gains=False
## output resolution (S2 only 10, 20, or 60 m)
s2_target_res=60
ancillary_data=True
#glint_correction=False
dsf_residual_glint_correction=True
dsf_residual_glint_correction_method=default
l2w_mask_water_parameters=True
l2w_mask_cirrus=True
l2w_mask_high_toa=True
l2w_mask_smooth=True
mask_smooth_sigma=3
merge_tiles=False
l2w_mask_negative_rhow=True
l2w_mask=True
rgb_rhot=False
rgb_rhos=False
map_l2w=False
map_l2r=False
map_title=False
map_colorbar=False
##map_colorbar_orientation=vertical
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
l2w_export_geotiff=True
Re: Acolite DSF results over Barnets Sea
Hi Asim
If you get mostly NaNs the aot you give is clearly too high. Perhaps a smaller increase is more appropriate than x2 or x1.5, e.g. +0.01 or +0.02
The code should not take longer with a user provided aot. With the dsf_aot_estimate=fixed option, by default the per pixel geometry is used, which will take longer as the atmospheric parameters are interpolated per pixel rather than across the scene. You can disable this (resolved_geometry=False) but it may give worse results.
Otherwise you can be a bit more patient or optimise the code.
Quinten
If you get mostly NaNs the aot you give is clearly too high. Perhaps a smaller increase is more appropriate than x2 or x1.5, e.g. +0.01 or +0.02
The code should not take longer with a user provided aot. With the dsf_aot_estimate=fixed option, by default the per pixel geometry is used, which will take longer as the atmospheric parameters are interpolated per pixel rather than across the scene. You can disable this (resolved_geometry=False) but it may give worse results.
Otherwise you can be a bit more patient or optimise the code.
Quinten
Re: Acolite DSF results over Barnets Sea
Thanks a lot! I hope I can tune Acolite for the high latitude waters. 
Regards
Asim
Regards
Asim
Re: Acolite DSF results over Barnets Sea
Hi,
I am still getting NaN values by doing aot_550 + 0.05. The settings file is :
inputfile=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Matchups/scenes/scene/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
output=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Acolite_MSI/Matchups_2021/Acolite/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
##limit=
mid_lat=
mid_lon=
l2w_parameters=Rrs_*, chl_oc2, chl_oc3
l2w_export_geotiff=True
aerosol_correction=dark_spectrum
dsf_aot_estimate=fixed
dsf_fixed_aot=0.21080622 (original dsf_fixed_aot= 0.1608)
dsf_fixed_lut=ACOLITE-LUT-202102-MOD1
dsf_write_aot_550 = True
gains=False
## output resolution (S2 only 10, 20, or 60 m)
s2_target_res=60
ancillary_data=True
#glint_correction=False
dsf_residual_glint_correction=True
dsf_residual_glint_correction_method=default
l2w_mask_water_parameters=True
l2w_mask_cirrus=True
l2w_mask_high_toa=True
l2w_mask_smooth=True
mask_smooth_sigma=3
merge_tiles=False
l2w_mask_negative_rhow=True
l2w_mask=True
rgb_rhot=False
rgb_rhos=False
map_l2w=False
map_l2r=False
map_title=False
map_colorbar=False
##map_colorbar_orientation=vertical
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
l2w_export_geotiff=True
I am still getting NaN values by doing aot_550 + 0.05. The settings file is :
inputfile=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Matchups/scenes/scene/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
output=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Acolite_MSI/Matchups_2021/Acolite/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
##limit=
mid_lat=
mid_lon=
l2w_parameters=Rrs_*, chl_oc2, chl_oc3
l2w_export_geotiff=True
aerosol_correction=dark_spectrum
dsf_aot_estimate=fixed
dsf_fixed_aot=0.21080622 (original dsf_fixed_aot= 0.1608)
dsf_fixed_lut=ACOLITE-LUT-202102-MOD1
dsf_write_aot_550 = True
gains=False
## output resolution (S2 only 10, 20, or 60 m)
s2_target_res=60
ancillary_data=True
#glint_correction=False
dsf_residual_glint_correction=True
dsf_residual_glint_correction_method=default
l2w_mask_water_parameters=True
l2w_mask_cirrus=True
l2w_mask_high_toa=True
l2w_mask_smooth=True
mask_smooth_sigma=3
merge_tiles=False
l2w_mask_negative_rhow=True
l2w_mask=True
rgb_rhot=False
rgb_rhos=False
map_l2w=False
map_l2r=False
map_title=False
map_colorbar=False
##map_colorbar_orientation=vertical
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
l2w_export_geotiff=True
Re: Acolite DSF results over Barnets Sea
It looks like the user-defined aot_550 is not working. I have tried it several times now.
Asim
Asim
Re: Acolite DSF results over Barnets Sea
I have now applied the same settings file as before but getting NaN values. I was not getting NaN values using the previous python version. Your help will be highly appreciated. Please note that not on all the scenes I am getting NaN values.
My setting file is :
## ACOLITE settings
## Written at 2021-11-04 15:33:49
dsf_aot_estimate=tiled
dsf_spectrum_option=intercept
dsf_wave_range=400,900
s2_target_res=60
resolved_geometry=True
gains_toa=1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
dsf_intercept_pixels=200
dsf_tile_dimensions=600,600
geometry_per_band=False
geometry_fixed_footprint=False
region_name=
limit=None
polygon=None
merge_tiles=False
merge_zones=True
extend_region=False
output=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Acolite_MSI/Matchups_2021/Acolite/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
output_geolocation=True
output_xy=False
output_geometry=True
output_rhorc=False
l8_orange_band=True
geometry_type=grids_footprint
geometry_res=60
smile_correction=True
use_tpg=True
inputfile_swir=None
pleiades_skip_pan=False
chris_interband_calibration=True
chris_noise_reduction=True
tact_run=False
tact_output_atmosphere=False
tact_output_intermediate=False
tact_map=True
verbosity=5
output_lt=False
blackfill_skip=True
blackfill_max=1.0
blackfill_wave=1600
copy_datasets=lon,lat,rhot_*
output_bt=False
l2w_parameters=Rrs_*,chl_oc2,chl_oc3
l2w_mask=True
l2w_mask_wave=1600
l2w_mask_threshold=0.0215
l2w_mask_water_parameters=True
l2w_mask_negative_rhow=True
l2w_mask_negative_wave_range=400,900
l2w_mask_cirrus=True
l2w_mask_cirrus_threshold=0.005
l2w_mask_cirrus_wave=1373
l2w_mask_high_toa=True
l2w_mask_high_toa_threshold=0.3
l2w_mask_smooth=True
l2w_mask_smooth_sigma=3
flag_exponent_swir=0
flag_exponent_cirrus=1
flag_exponent_toa=2
flag_exponent_negative=3
flag_exponent_outofscene=4
atmospheric_correction=True
aerosol_correction=dark_spectrum
min_tgas_aot=0.85
min_tgas_rho=0.75
ancillary_data=True
uoz_default=0.3
uwv_default=1.5
pressure=1013.25
elevation=None
dem_pressure=False
dem_pressure_resolved=True
dem_pressure_percentile=25.0
dem_pressure_write=False
dsf_interface_reflectance=False
dsf_interface_option=default
dsf_interface_lut=RSKY-202102
wind=None
wind_default=2.0
dsf_residual_glint_correction=True
dsf_residual_glint_correction_method=default
dsf_residual_glint_wave_range=1500,2400
glint_force_band=None
glint_mask_rhos_wave=1600
glint_mask_rhos_threshold=0.05
glint_write_rhog_ref=False
glint_write_rhog_all=False
dsf_percentile=1.0
dsf_filter_toa=False
dsf_filter_percentile=5.0
dsf_filter_box=5,5
dsf_filter_aot=False
dsf_smooth_aot=True
dsf_smooth_box=20,20
dsf_model_selection=min_drmsd
dsf_min_tile_cover=0.1
dsf_min_tile_aot=0.01
dsf_write_tiled_parameters=False
dsf_exclude_bands=,
dsf_write_aot_550=True
dsf_fixed_aot=None
dsf_fixed_lut=ACOLITE-LUT-202102-MOD2
exp_swir_threshold=0.0215
exp_fixed_epsilon=True
exp_fixed_epsilon_percentile=50.0
exp_fixed_aerosol_reflectance=True
exp_fixed_aerosol_reflectance_percentile=5.0
exp_wave1=1600
exp_wave2=2200
exp_alpha=None
exp_alpha_weighted=True
exp_epsilon=None
exp_gamma=None
exp_output_intermediate=False
gains=False
rgb_rhot=False
rgb_rhos=False
map_l2w=False
map_title=False
map_dpi=300
map_ext=png
map_limit=None
map_colorbar=False
map_colorbar_orientation=vertical
map_auto_range=False
map_auto_range_percentiles=1.0,99.0
map_fill_outrange=False
map_fill_color=LightGrey
map_default_colormap=viridis
rgb_red_wl=650
rgb_green_wl=560
rgb_blue_wl=480
rgb_min=0.0,0.0,0.0
rgb_max=0.15,0.15,0.15
map_projected=False
map_raster=False
map_pcolormesh=False
map_points=None
l1r_export_geotiff=False
l2t_export_geotiff=False
l2r_export_geotiff=False
l2w_export_geotiff=True
export_geotiff_coordinates=False
l1r_delete_netcdf=False
l2t_delete_netcdf=False
l2r_delete_netcdf=False
l2w_delete_netcdf=False
luts=ACOLITE-LUT-202102-MOD1,ACOLITE-LUT-202102-MOD2
slicing=False
inputfile=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Matchups/scenes/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
mid_lat=
mid_lon=
glint_correction=True
mask_smooth_sigma=3
map_l2r=False
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
runid=20211104_153307
My setting file is :
## ACOLITE settings
## Written at 2021-11-04 15:33:49
dsf_aot_estimate=tiled
dsf_spectrum_option=intercept
dsf_wave_range=400,900
s2_target_res=60
resolved_geometry=True
gains_toa=1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
dsf_intercept_pixels=200
dsf_tile_dimensions=600,600
geometry_per_band=False
geometry_fixed_footprint=False
region_name=
limit=None
polygon=None
merge_tiles=False
merge_zones=True
extend_region=False
output=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Acolite_MSI/Matchups_2021/Acolite/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
output_geolocation=True
output_xy=False
output_geometry=True
output_rhorc=False
l8_orange_band=True
geometry_type=grids_footprint
geometry_res=60
smile_correction=True
use_tpg=True
inputfile_swir=None
pleiades_skip_pan=False
chris_interband_calibration=True
chris_noise_reduction=True
tact_run=False
tact_output_atmosphere=False
tact_output_intermediate=False
tact_map=True
verbosity=5
output_lt=False
blackfill_skip=True
blackfill_max=1.0
blackfill_wave=1600
copy_datasets=lon,lat,rhot_*
output_bt=False
l2w_parameters=Rrs_*,chl_oc2,chl_oc3
l2w_mask=True
l2w_mask_wave=1600
l2w_mask_threshold=0.0215
l2w_mask_water_parameters=True
l2w_mask_negative_rhow=True
l2w_mask_negative_wave_range=400,900
l2w_mask_cirrus=True
l2w_mask_cirrus_threshold=0.005
l2w_mask_cirrus_wave=1373
l2w_mask_high_toa=True
l2w_mask_high_toa_threshold=0.3
l2w_mask_smooth=True
l2w_mask_smooth_sigma=3
flag_exponent_swir=0
flag_exponent_cirrus=1
flag_exponent_toa=2
flag_exponent_negative=3
flag_exponent_outofscene=4
atmospheric_correction=True
aerosol_correction=dark_spectrum
min_tgas_aot=0.85
min_tgas_rho=0.75
ancillary_data=True
uoz_default=0.3
uwv_default=1.5
pressure=1013.25
elevation=None
dem_pressure=False
dem_pressure_resolved=True
dem_pressure_percentile=25.0
dem_pressure_write=False
dsf_interface_reflectance=False
dsf_interface_option=default
dsf_interface_lut=RSKY-202102
wind=None
wind_default=2.0
dsf_residual_glint_correction=True
dsf_residual_glint_correction_method=default
dsf_residual_glint_wave_range=1500,2400
glint_force_band=None
glint_mask_rhos_wave=1600
glint_mask_rhos_threshold=0.05
glint_write_rhog_ref=False
glint_write_rhog_all=False
dsf_percentile=1.0
dsf_filter_toa=False
dsf_filter_percentile=5.0
dsf_filter_box=5,5
dsf_filter_aot=False
dsf_smooth_aot=True
dsf_smooth_box=20,20
dsf_model_selection=min_drmsd
dsf_min_tile_cover=0.1
dsf_min_tile_aot=0.01
dsf_write_tiled_parameters=False
dsf_exclude_bands=,
dsf_write_aot_550=True
dsf_fixed_aot=None
dsf_fixed_lut=ACOLITE-LUT-202102-MOD2
exp_swir_threshold=0.0215
exp_fixed_epsilon=True
exp_fixed_epsilon_percentile=50.0
exp_fixed_aerosol_reflectance=True
exp_fixed_aerosol_reflectance_percentile=5.0
exp_wave1=1600
exp_wave2=2200
exp_alpha=None
exp_alpha_weighted=True
exp_epsilon=None
exp_gamma=None
exp_output_intermediate=False
gains=False
rgb_rhot=False
rgb_rhos=False
map_l2w=False
map_title=False
map_dpi=300
map_ext=png
map_limit=None
map_colorbar=False
map_colorbar_orientation=vertical
map_auto_range=False
map_auto_range_percentiles=1.0,99.0
map_fill_outrange=False
map_fill_color=LightGrey
map_default_colormap=viridis
rgb_red_wl=650
rgb_green_wl=560
rgb_blue_wl=480
rgb_min=0.0,0.0,0.0
rgb_max=0.15,0.15,0.15
map_projected=False
map_raster=False
map_pcolormesh=False
map_points=None
l1r_export_geotiff=False
l2t_export_geotiff=False
l2r_export_geotiff=False
l2w_export_geotiff=True
export_geotiff_coordinates=False
l1r_delete_netcdf=False
l2t_delete_netcdf=False
l2r_delete_netcdf=False
l2w_delete_netcdf=False
luts=ACOLITE-LUT-202102-MOD1,ACOLITE-LUT-202102-MOD2
slicing=False
inputfile=/projects/NS9530K/nansen_legacy/work/CIRFA_Work/L8_S2_Data_Evaluation/Acolite_Dataset/Matchups/scenes/S2B_MSIL1C_20210621T112629_N0300_R080_T33XXF_20210621T121350.SAFE
mid_lat=
mid_lon=
glint_correction=True
mask_smooth_sigma=3
map_l2r=False
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
runid=20211104_153307
Re: Acolite DSF results over Barnets Sea
Hi Asim
You can check the content of the l2_flags dataset. For the scene S2A_MSIL1C_20210621T121651_N0300_R009_T33XXF_20210621T160320 there are a lot of "8" bits set. The 8 bit is only set when negatives are retrieved (flag_exponent_negative=3). You can disable this masking by setting l2w_mask_negative_rhow=False but keep in mind that negative reflectances may not be what you need.
Quinten
You can check the content of the l2_flags dataset. For the scene S2A_MSIL1C_20210621T121651_N0300_R009_T33XXF_20210621T160320 there are a lot of "8" bits set. The 8 bit is only set when negatives are retrieved (flag_exponent_negative=3). You can disable this masking by setting l2w_mask_negative_rhow=False but keep in mind that negative reflectances may not be what you need.
Quinten
Re: Acolite DSF results over Barnets Sea
For the user defined aot, you should put only the aot value on the line, without comment: e.g. dsf_fixed_aot=0.21080622
Quinten
Quinten