Why are my data masked or not masked?
ACOLITE produces three kinds of NetCDF files:
- L1R, containing the top-of-atmosphere reflectance (rhot_*) as converted to the ACOLITE format from the sensor specific L1 files
- L2R, containing the top-of-atmosphere reflectance (rhot_*) and the surface-level reflectance after atmospheric correction (rhos_*)
- L2W, containing user requested parameters, e.g. water-leaving radiance reflectance (rhow_*), Remote sensing reflectance (Rrs_*), or outputs from any of the included parameter retrieval algorithms
No quality flags or masking is applied to the L1R and L2R files. If you use these files you may need to be careful and confirm you are looking at what you think you are looking at! There may be sun glint, haze, or cirrus clouds, conditions which contaminate the estimate of surface-level reflectance by ACOLITE/DSF. You can see examples of slightly modified tests on these products e.g. in Vanhellemont 2019 (https://doi.org/10.1016/j.rse.2019.03.010) Vanhellemont 2020 (https://doi.org/10.1364/OE.397456) and Vanhellemont and Ruddick 2021 (https://doi.org/10.1016/j.rse.2021.112284)
The "water parameters" in the L2W files are masked using a simple set of thresholds and tests, the results of which are stored as binary flags in the l2_flags dataset. By default five tests are performed, each with an associated bit flag. Pixel values in the l2_flags can have any combination of these, from 0 (no flags set) to 31 (all 5 flags set).
Performed tests and the meaning of l2_flags values
- NIR or SWIR threshold test, stored as bit flag 0, i.e. 2^0 or integer value 1. This is a threshold test (set by l2w_mask_threshold) on a NIR or SWIR band depending on availability (set by l2w_mask_wave). If this bit is set the rhot_* in the selected band exceeded the threshold. Can be disabled by setting l2w_mask=False
- CIRRUS threshold test, stored as bit flag 1, i.e. 2^1 or integer value 2. This is a threshold test (l2w_mask_cirrus_threshold) on a cirrus band at around 1.37 micron (set by l2w_mask_cirrus_wave). If this bit is set the rhot_* in the cirrus band exceeded the threshold. Not all sensors have such a band. Can be disabled by setting l2w_mask_cirrus=False
- TOA threshold test, stored as bit flag 2, i.e. 2^2 or integer value 4. This is a threshold test (l2w_mask_high_toa_threshold) on all bands of a sensor, rejecting the brightest pixels. If this bit is set the rhot_* in any band exceeded the threshold. Can be disabled by setting l2w_mask_high_toa=False
- NEGATIVE rhos_* test, stored as bit flag 3, i.e. 2^3 or integer value 8. This is a test whether negatives are retrieved in the surface-level reflectance for bands within a specified wavelength (set by l2w_mask_negative_wave_range). The DSF should be reasonably robust to retrieving negatives but in some cases they occur and cannot be avoided (e.g. if the observed rhot_* is lower than the computed Rayleigh reflectance). Can be disabled by setting l2w_mask_negative_rhow=False
- EXTENT test, stored as bit flag 4, i.e. 2^4 or integer value 16. This flag is set for scenes that contain pixels outside the satellite swath, e.g. in projected Sentinel-2 or Landsat data. Cannot currently be disabled.
Examples
The l2_flags combines these values per pixel, but due to the use of binary masks they can be uniquely identified. Any l2_flags value of 8 or greater will have failed the negative surface-level reflectance test. A pixel with l2_flags of 7 will fail the first three tests (integer values 1+2+4), for a sensor without cirrus band a similar target will give l2_flags of 5 (integer values 1+4). The last two examples mean that the pixel has high reflectance and will likely be non-water.
Additional notes
The masks are in place to try and provide good quality L2W products. In general you should only disable the masks if you understand what they mean, why they are in place, and if you use the resulting products with extra care. If you disable masks you may get additional outputs, but they are likely of worse quality! Don't be surprised if you get negative reflectances if you set l2w_mask_negative_rhow=False!
If your L2R rhos_* data are NaN this causes the L2W to be NaN as well. This only occurs for high aerosol optical depths when the LUT dimension reduction is set. If your rhos_* are NaN please check this post on how to resolve it.