Hi Quinten-
I’ve got an issue similar to that above (shape mismatch at the start of L2 processing). The acolite version I’m using automatically retrieves LUTs (per your last post in this thread).
I’ve listed a S2B tile (from ESA directly) in the example below, but I get the same error for data I’ve acquired from USGS (S2, Landsat8, Landsat7). Processing completes, however, for Landsat8 data acquired from AWS or gsutil. Any advice?
-brian
> wget --content-disposition --continue --user={UN} --password={PW} "https://scihub.copernicus.eu/dhus/odata/v1/Products('f1d7015e-fa08-43ab-b1ee-d1a1879fcf01')/\$value"
> unzip S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.zip
> conda activate ACOLITE_BBB
#[Note, this conda env was created as specified in the acolite git repo].
> python3 /shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py --cli --settings /work/b/bbarnes4/processing/working3/E2021160160509/acolite.cfg
Running generic ACOLITE processing - Generic Version Public Beta - April 2021
Run ID - 20210615_120315
/work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
/work/b/bbarnes4/processing/completed/E2021160160509.L2/acolite_run_20210615_120315_l1r_settings.txt
Starting conversion of 1 scenes
Starting conversion of /work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
Importing metadata from L1C_T17RNJ_A022245_20210609T160548
Reading per pixel geometry
Computing per detector geometry
Detector 2
/shares/cms_optics/apps/acolite_20210114/acolite/acolite/sentinel2/l1_convert.py:359: RuntimeWarning: Mean of empty slice
ave_vza = np.nanmean(ave_vza, axis=2)
/shares/cms_optics/apps/acolite_20210114/acolite/acolite/sentinel2/l1_convert.py:360: RuntimeWarning: Mean of empty slice
ave_vaa = np.nanmean(ave_vaa, axis=2)
Detector 6
Detector 7
Detector 4
Detector 5
Detector 3
Wrote raa
Wrote vza
Wrote sza
Writing geolocation lon/lat
(2232, 3033)
Wrote lon
(2232, 3033)
Wrote lat
Converting bands
Converting bands: Wrote rhot_442 ((2232, 3033))
...
Converting bands: Wrote rhot_2186 ((2232, 3033))
Conversion took 17.6 seconds
Created /work/b/bbarnes4/processing/completed/E2021160160509.L2/S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc
Running acolite for /work/b/bbarnes4/processing/completed/E2021160160509.L2/S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc
Scene too recent to get ancillary data
Using DSF atmospheric correction
Processing with 24 tiles (4x6)
Writing lon
Wrote lon
Writing lat
Wrote lat
1 rhot_442
Traceback (most recent call last):
File "/shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py", line 64, in <module>
launch_acolite()
File "/shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py", line 58, in launch_acolite
ac.acolite.acolite_run(args.settings, inputfile=inputfile, output=output)
File "/shares/cms_optics/apps/acolite_20210114/acolite/acolite/acolite/acolite_run.py", line 108, in acolite_run
ret = ac.acolite.acolite_l2r(l1r, settings = setu, verbosity = verbosity)
File "/shares/cms_optics/apps/acolite_20210114/acolite/acolite/acolite/acolite_l2r.py", line 414, in acolite_l2r
tmp = lutdw[lut]['rgi'][b]((gem.data_mem['pressure'+gk],
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/scipy/interpolate/interpolate.py", line 2496, in __call__
xi = _ndim_coords_from_arrays(xi, ndim=ndim)
File "interpnd.pyx", line 157, in scipy.interpolate.interpnd._ndim_coords_from_arrays
File "interpnd.pyx", line 168, in scipy.interpolate.interpnd._ndim_coords_from_arrays
File "<__array_function__ internals>", line 5, in broadcast_arrays
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/numpy/lib/stride_tricks.py", line 538, in broadcast_arrays
shape = _broadcast_shape(*args)
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/numpy/lib/stride_tricks.py", line 420, in _broadcast_shape
b = np.broadcast(*args[:32])
ValueError: shape mismatch: objects cannot be broadcast to a single shape
>>> print(ac.config['data_dir'])
/shares/cms_optics/apps/acolite_20210114/acolite/data
>>> print(rsrd.keys())
dict_keys(['PlanetScope_0d06', 'L8_OLI', 'WorldView3', 'PHR1A', 'L5_TM', 'VENµS_VSSC', 'S2B_MSI', 'RapidEye', 'PlanetScope_0d05', 'PlanetScope_0e', 'L7_ETM', 'WorldView2', 'PHR1B', 'L8_TIRS', 'SPOT6', 'L8_OLI_ORANGE', 'PlanetScope_0f', 'L7_ETM_B6', 'PlanetScope_0c', 'S3A_OLCI', 'SPOT7', 'S3B_OLCI', 'L5_TM_B6', 'PlanetScope_22', 'S2A_MSI'])
>>> print(ac.shared.nc_gatts(l1r_file))
{'generated_by': 'ACOLITE', 'generated_on': '2021-06-15 12:03:20 EDT', 'contact': 'Quinten Vanhellemont', 'product_type': 'NetCDF', 'metadata_profile': 'beam', 'metadata_version': '0.5', 'auto_grouping': 'rhot:rhorc:rhos:rhow:Rrs', 'Conventions': 'CF-1.7', 'sensor': 'S2B_MSI', 'isodate': '2021-06-09T16:06:14.435177+00:00', 'global_dims': array([2232, 3033]), 'sza': 17.2653337075683, 'vza': 4.752444361204014, 'raa': 6.171068634648037, 'se_distance': 1.0151026255835212, 'mus': 0.9549405487090699, 'granule': 'L1C_T17RNJ_A022245_20210609T160548', 'mgrs_tile': 'T17RNJ', 'acolite_file_type': 'L1R', 'tile_code': 'T17RNJ', 'oname': 'S2B_MSI_2021_06_09_16_06_14_T17RNJ', 'ofile': '/work/b/bbarnes4/processing/completed/E2021160160509.L2/S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc', '1_wave': 442.2312743651042, '1_name': '442', '1_f0': 18739.930412260506, '2_wave': 492.1349288237099, '2_name': '492', '2_f0': 19597.906236233655, '3_wave': 558.9524402572151, '3_name': '559', '3_f0': 18249.439122035074, '4_wave': 664.9400107071172, '4_name': '665', '4_f0': 15128.02312147004, '5_wave': 703.8279778656998, '5_name': '704', '5_f0': 14254.082494018856, '6_wave': 739.1263016544829, '6_name': '739', '6_f0': 12907.275347038803, '7_wave': 779.7278300593489, '7_name': '780', '7_f0': 11754.386815639207, '8_wave': 832.9490246389458, '8_name': '833', '8_f0': 10411.810614390462, '8A_wave': 863.9808438697365, '8A_name': '864', '8A_f0': 9537.54087863298, '9_wave': 943.1826634691346, '9_name': '943', '9_f0': 8176.350245943502, '10_wave': 1376.8832493303946, '10_name': '1377', '10_f0': 3653.9462660045874, '11_wave': 1610.419203764768, '11_name': '1610', '11_f0': 2470.7582160831216, '12_wave': 2185.699149849077, '12_name': '2186', '12_f0': 877.4766514889386, 'scene_xrange': array([499980., 609770.]), 'scene_yrange': array([2900040., 2790250.]), 'scene_proj4_string': '+proj=utm +zone=17 +datum=WGS84 +units=m +no_defs ', 'scene_pixel_size': array([ 10., -10.]), 'scene_dims': array([10980, 10980]), 'sub': array([6034, 8496, 3033, 2232]), 'limit': array([ 25.25, -80.4 , 25.45, -80.1 ]), 'xrange': array([560320., 590640.]), 'yrange': array([2815080., 2792770.]), 'proj4_string': '+proj=utm +zone=17 +datum=WGS84 +units=m +no_defs ', 'pixel_size': array([ 10., -10.])}
> cat /work/b/bbarnes4/processing/working3/E2021160160509/acolite.cfg
inputfile=/work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
## settings for cropping
region_name=
limit=25.25,-80.4,25.45,-80.1
polygon=None
merge_tiles=False
merge_zones=True
extend_region=False
## new output settings
output_geolocation=True
output_xy=False
output_geometry=True
output_rhorc=False
## Sentinel-2 per pixel geometry
geometry_type=grids_footprint
geometry_res=60
## Sentinel-3 options
smile_correction=True
use_tpg=True
## WorldView only (optional)
inputfile_swir=None
## TACT settings
tact_run=False
tact_output_atmosphere=False
tact_output_intermediate=False
tact_map=True
## printout verbosity
verbosity=5
## test cropped scenes for blackfill coverage
blackfill_skip=True
blackfill_max=1.0
blackfill_wave=1600
## copy datasets from L1R to L2R
copy_datasets=lon,lat
## output l2w parameters
l2w_parameters=Rrs_*,rhorc_*,chl_oc3,fai
l2w_mask=True
l2w_mask_wave=1600
l2w_mask_threshold=0.0215
l2w_mask_water_parameters=True
l2w_mask_negative_rhow=True
l2w_mask_negative_wave_range=400,900
l2w_mask_cirrus=True
l2w_mask_cirrus_threshold=0.005
l2w_mask_cirrus_wave=1373
l2w_mask_high_toa=True
l2w_mask_high_toa_threshold=0.3
## smooth datasets before masking (get rid of some speckle in the masks)
l2w_mask_smooth=True
l2w_mask_smooth_sigma=3
## flag settings
flag_exponent_swir=0
flag_exponent_cirrus=1
flag_exponent_toa=2
flag_exponent_negative=3
flag_exponent_outofscene=4
## atmospheric correction settings
aerosol_correction=dark_spectrum
## minimum gas transmittance to allow aot retrieval in that band
min_tgas_aot=0.85
## minimum gas transmittance to allow surface reflectance retrieval in that band
min_tgas_rho=0.75
## generic options
ancillary_data=True
gas_transmittance=True
uoz_default=0.3
uwv_default=1.5
pressure=None
elevation=None
lut_pressure=True
## DEM options
dem_pressure=False
dem_pressure_resolved=True
dem_pressure_percentile=25
dem_pressure_write=False
## correction for sky reflectance on water surface
sky_correction=True
sky_correction_option=default
sky_correction_lut=RSKY-202102
wind=2
## glint correction
glint_correction=False
glint_force_band=None
glint_mask_rhos_wave=1600
glint_mask_rhos_threshold=0.05
glint_write_rhog_ref=False
glint_write_rhog_all=False
## dark spectrum fitting options
dsf_path_reflectance=tiled
## options darkest, percentile, intercept
dsf_spectrum_option=intercept
dsf_percentile=1
dsf_intercept_pixels=1000
## apply median filter to TOA data
dsf_filter_toa=False
dsf_filter_percentile=5
dsf_filter_box=5,5
dsf_filter_aot=False
# smooth retrieved aot for resolved processing
dsf_smooth_aot=True
dsf_smooth_box=20,20
#
dsf_full_scene=False
dsf_model_selection=min_drmsd
#dsf_list_selection=intercept
#dsf_tile_dimensions=None
dsf_min_tile_cover=0.10
dsf_min_tile_aot=0.01
dsf_plot_retrieved_tiles=True
dsf_plot_dark_spectrum=True
dsf_write_tiled_parameters=False
dsf_wave_range=400,2500
dsf_exclude_bands=,
dsf_write_aot_550=False
extra_ac_parameters=False
## fixed aot option
dsf_fixed_aot=None
dsf_fixed_lut=ACOLITE-LUT-202102-MOD2
## exponential options
exp_swir_threshold=0.0215
exp_fixed_epsilon=True
exp_fixed_epsilon_percentile=50
exp_fixed_aerosol_reflectance=True
exp_fixed_aerosol_reflectance_percentile=5
exp_wave1=1600
exp_wave2=2200
exp_alpha=None
exp_alpha_weighted=True
exp_epsilon=None
exp_gamma=None
exp_output_intermediate=False
##
s2_target_res=10
l8_output_bt=False
l8_output_lt_tirs=False
resolved_geometry=False
resolved_geometry_write=False
## output of xy easting and northing
output=/work/b/bbarnes4/processing/completed/E2021160160509.L2
## gain settings
gains=False
gains_toa=None
# output RGB / L2W maps
rgb_rhot=False
rgb_rhos=True
map_l2w=False
# mapping options
map_title=True
map_dpi=300
map_ext=png
limit=25.25,-80.4,25.45,-80.1
## map colour bar options
map_colorbar=False
map_colorbar_orientation=vertical
map_auto_range=False
map_auto_range_percentiles=1,99
map_fill_outrange=False
map_fill_color=LightGrey
map_default_colormap=viridis
# RGB options
rgb_red_wl=650
rgb_green_wl=560
rgb_blue_wl=480
rgb_min=0.0,0.0,0.0
rgb_max=0.15,0.15,0.15
rgb_pan_sharpen=False
# more mapping options for 'projected image'
map_projected=False
map_raster=True
map_pcolormesh=False
map_scalebar=True
map_scalepos=UL
map_scalecolor=White
map_scalecolor_rgb=White
map_scalelen=None
map_colorbar_edge=True
map_max_dim=1000
map_points=None
# NetCDF output options
l1r_nc_compression=True
l2r_nc_compression=True
l2w_nc_compression=True
# NetCDF deletion options
l1r_nc_delete=False
l2r_nc_delete=False
l2w_nc_delete=False
# Convert reflectance floats to scaled integers
l2w_nc_scaled=False
l2w_nc_scaled_offset=0.02
l2w_nc_scaled_factor=0.0001
# GeoTIFF export options
l2r_export_geotiff=False
l2w_export_geotiff=False
export_geotiff_coordinates=False
# LUTs
luts=ACOLITE-LUT-202102-MOD1, ACOLITE-LUT-202102-MOD2
slicing=False
ValueError: shape mismatch: objects cannot be broadcast to a single shape
Re: Acolite failing to run on Sentinel-2 after generating LUTs
Hi Brian
Thanks for your question and detailed report!
I've been able to process files from EE fine for your ROI on two systems (Mac and Linux). The only issue I encountered was the outputting of rhorc products to L2W and generating them in tiled processing mode - should be fixed in the last commits.
Perhaps it is a dependency version issue. Did you recently make the ACOLITE_BBB environment?
Quinten
Thanks for your question and detailed report!
I've been able to process files from EE fine for your ROI on two systems (Mac and Linux). The only issue I encountered was the outputting of rhorc products to L2W and generating them in tiled processing mode - should be fixed in the last commits.
Perhaps it is a dependency version issue. Did you recently make the ACOLITE_BBB environment?
Quinten
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
I used the following minimum settings file:
Quinten
Code: Select all
output=/path/to/output/
inputfile=/path/to/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE
limit=25.25,-80.4,25.45,-80.1
l2w_parameters=Rrs_*,rhorc_*,chl_oc3,fai
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Quinten-
Thanks for looking into this. The conda env was created ~1 week ago (details below). I retried running the with your minimum settings file. Processing gets further (I think because it’s using ReverseLUTs path now) but still craps out with a similar error.
-brian
> cat acolite_basic.cfg
output=/work/b/bbarnes4/processing/completed/E2021160160509.L2_basic/
inputfile=/work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
limit=25.25,-80.4,25.45,-80.1
l2w_parameters=Rrs_*,rhorc_*,chl_oc3,fai
> python /shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py --cli --settings /work/b/bbarnes4/processing/working3/E2021160160509/acolite_basic.cfg
Running generic ACOLITE processing - Generic Version Public Beta - April 2021
Run ID - 20210616_095807
/work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
/work/b/bbarnes4/processing/completed/E2021160160509.L2_basic//acolite_run_20210616_095807_l1r_settings.txt
Starting conversion of 1 scenes
Starting conversion of /work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
Importing metadata from L1C_T17RNJ_A022245_20210609T160548
Reading per pixel geometry
Computing per detector geometry
Detector 2
/shares/cms_optics/apps/acolite_20210114/acolite/acolite/sentinel2/l1_convert.py:359: RuntimeWarning: Mean of empty slice
ave_vza = np.nanmean(ave_vza, axis=2)
/shares/cms_optics/apps/acolite_20210114/acolite/acolite/sentinel2/l1_convert.py:360: RuntimeWarning: Mean of empty slice
ave_vaa = np.nanmean(ave_vaa, axis=2)
Detector 6
Detector 7
Detector 4
Detector 5
Detector 3
Wrote raa
Wrote vza
Wrote sza
Writing geolocation lon/lat
(2232, 3033)
Wrote lon
(2232, 3033)
Wrote lat
Converting bands
Converting bands: Wrote rhot_442 ((2232, 3033))
Converting bands: Wrote rhot_492 ((2232, 3033))
Converting bands: Wrote rhot_559 ((2232, 3033))
Converting bands: Wrote rhot_665 ((2232, 3033))
Converting bands: Wrote rhot_704 ((2232, 3033))
Converting bands: Wrote rhot_739 ((2232, 3033))
Converting bands: Wrote rhot_780 ((2232, 3033))
Converting bands: Wrote rhot_833 ((2232, 3033))
Converting bands: Wrote rhot_864 ((2232, 3033))
Converting bands: Wrote rhot_943 ((2232, 3033))
Converting bands: Wrote rhot_1377 ((2232, 3033))
Converting bands: Wrote rhot_1610 ((2232, 3033))
Converting bands: Wrote rhot_2186 ((2232, 3033))
Conversion took 16.9 seconds
Created /work/b/bbarnes4/processing/completed/E2021160160509.L2_basic//S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc
Running acolite for /work/b/bbarnes4/processing/completed/E2021160160509.L2_basic//S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
Using DSF atmospheric correction
Processing with 24 tiles (4x6)
Writing lon
Wrote lon
Writing lat
Wrote lat
1 rhot_442
S2B_MSI/B1 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B1 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
2 rhot_492
S2B_MSI/B2 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B2 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
3 rhot_559
S2B_MSI/B3 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B3 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
4 rhot_665
S2B_MSI/B4 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B4 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
5 rhot_704
S2B_MSI/B5 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B5 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
6 rhot_739
S2B_MSI/B6 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B6 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
7 rhot_780
S2B_MSI/B7 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B7 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
8 rhot_833
S2B_MSI/B8 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B8 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
8A rhot_864
S2B_MSI/B8A ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B8A ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
Choosing best fitting model: min_drmsd
Computing RMSD for model ACOLITE-LUT-202102-MOD1
Computing RMSD for model ACOLITE-LUT-202102-MOD2
use_revlut True
Wrote rhot_442
Computing surface reflectance 1 442 0.998
Interpolating tiles
Traceback (most recent call last):
File "/shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py", line 64, in <module>
launch_acolite()
File "/shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py", line 58, in launch_acolite
ac.acolite.acolite_run(args.settings, inputfile=inputfile, output=output)
File "/shares/cms_optics/apps/acolite_20210114/acolite/acolite/acolite/acolite_run.py", line 108, in acolite_run
ret = ac.acolite.acolite_l2r(l1r, settings = setu, verbosity = verbosity)
File "/shares/cms_optics/apps/acolite_20210114/acolite/acolite/acolite/acolite_l2r.py", line 913, in acolite_l2r
cur_rhorc = (cur_rhorc - rorayl_cur) / (dtotr_cur*utotr_cur)
ValueError: operands could not be broadcast together with shapes (2232,3033) (4,6)
> ll -d /home/b/bbarnes4/.anaconda3/envs/ACOLITE_BBB/
drwxrwxr-x 24 bbarnes4 usfuser 4096 Jun 10 11:47 /home/b/bbarnes4/.anaconda3/envs/ACOLITE_BBB/
> conda list --name ACOLITE_BBB
# packages in environment at /home/b/bbarnes4/.anaconda3/envs/ACOLITE_BBB:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
alsa-lib 1.2.3 h516909a_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
astropy 4.2.1 py39hce5d2b2_0 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
boost-cpp 1.74.0 hc6e9bd1_3 conda-forge
brotli 1.0.9 h9c3ff4c_4 conda-forge
brotlipy 0.7.0 py39h3811e60_1001 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h7f98852_1 conda-forge
ca-certificates 2021.5.30 ha878542_0 conda-forge
cairo 1.16.0 h6cf1ce9_1008 conda-forge
cartopy 0.19.0.post1 py39h3b23250_0 conda-forge
certifi 2021.5.30 py39hf3d152e_0 conda-forge
cffi 1.14.5 py39he32792d_0 conda-forge
cfitsio 3.470 hb418390_7 conda-forge
cftime 1.5.0 py39hce5d2b2_0 conda-forge
chardet 4.0.0 py39hf3d152e_1 conda-forge
charls 2.2.0 h9c3ff4c_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
configobj 5.0.6 py_0 conda-forge
cryptography 3.4.7 py39hbca0aa6_0 conda-forge
curl 7.77.0 hea6ffbf_0 conda-forge
cycler 0.10.0 py_2 conda-forge
cytoolz 0.11.0 py39h3811e60_3 conda-forge
dask-core 2021.6.0 pyhd8ed1ab_0 conda-forge
dbus 1.13.6 h48d8840_2 conda-forge
decorator 5.0.9 pyhd8ed1ab_0 conda-forge
expat 2.4.1 h9c3ff4c_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
fsspec 2021.6.0 pyhd8ed1ab_0 conda-forge
gdal 3.2.1 py39h409cc32_7 conda-forge
geos 3.9.1 h9c3ff4c_2 conda-forge
geotiff 1.6.0 h2b14fbe_4 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glib 2.68.2 h9c3ff4c_2 conda-forge
glib-tools 2.68.2 h9c3ff4c_2 conda-forge
gst-plugins-base 1.18.4 hf529b03_2 conda-forge
gstreamer 1.18.4 h76c114f_2 conda-forge
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
icu 68.1 h58526e2_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagecodecs 2021.3.31 py39h559889c_0 conda-forge
imageio 2.9.0 py_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
kealib 1.4.14 hcc255d8_2 conda-forge
kiwisolver 1.3.1 py39h1a9c180_1 conda-forge
krb5 1.19.1 hcc1bbae_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge
lerc 2.2.1 h9c3ff4c_0 conda-forge
libaec 1.0.4 h9c3ff4c_1 conda-forge
libblas 3.9.0 9_openblas conda-forge
libcblas 3.9.0 9_openblas conda-forge
libclang 11.1.0 default_ha53f305_1 conda-forge
libcurl 7.77.0 h2574ce0_0 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.7 h7f98852_5 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libffi 3.3 h58526e2_2 conda-forge
libgcc-ng 9.3.0 h2828fa1_19 conda-forge
libgdal 3.2.1 h38ff51b_7 conda-forge
libgfortran-ng 9.3.0 hff62375_19 conda-forge
libgfortran5 9.3.0 hff62375_19 conda-forge
libglib 2.68.2 h3e27bee_2 conda-forge
libgomp 9.3.0 h2828fa1_19 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h238a007_1013 conda-forge
liblapack 3.9.0 9_openblas conda-forge
libllvm11 11.1.0 hf817b99_2 conda-forge
libnetcdf 4.7.4 nompi_h56d31a8_107 conda-forge
libnghttp2 1.43.0 h812cca2_0 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.15 pthreads_h8fe5266_1 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 13.3 hd57d9b9_0 conda-forge
librttopo 1.1.0 h1185371_6 conda-forge
libspatialite 5.0.1 he52d314_3 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 h6de172a_19 conda-forge
libtiff 4.2.0 hbd63e13_2 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxcb 1.13 h7f98852_1003 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.12 h72842e0_0 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
matplotlib 3.4.2 py39hf3d152e_0 conda-forge
matplotlib-base 3.4.2 py39h2fa2bec_0 conda-forge
mysql-common 8.0.23 ha770c72_2 conda-forge
mysql-libs 8.0.23 h935591d_2 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
netcdf4 1.5.6 nompi_py39h36800e2_102 conda-forge
networkx 2.5 py_0 conda-forge
nspr 4.30 h9c3ff4c_0 conda-forge
nss 3.66 hb5efdd6_0 conda-forge
numpy 1.20.3 py39hdbf815f_1 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
packaging 20.9 pyh44b312d_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pillow 8.2.0 py39hf95b381_1 conda-forge
pip 21.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.3.0 pyhd8ed1ab_0 conda-forge
poppler 0.89.0 h2de54a5_5 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 13.3 h2510834_0 conda-forge
proj 7.2.0 h277dcde_2 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyerfa 1.7.3 py39hce5d2b2_1 conda-forge
pyhdf 0.10.3 py39hb0d3d0c_0 conda-forge
pykdtree 1.3.4 py39hce5d2b2_0 conda-forge
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyproj 3.0.1 py39h50a64a9_0 conda-forge
pyqt 5.12.3 py39hf3d152e_7 conda-forge
pyqt-impl 5.12.3 py39h0fcd23e_7 conda-forge
pyqt5-sip 4.19.18 py39he80948d_7 conda-forge
pyqtchart 5.12 py39h0fcd23e_7 conda-forge
pyqtwebengine 5.12.1 py39h0fcd23e_7 conda-forge
pyresample 1.20.0 py39hde0f152_0 conda-forge
pyshp 2.1.3 pyh44b312d_0 conda-forge
pysocks 1.7.1 py39hf3d152e_3 conda-forge
python 3.9.4 hffdb5ce_0_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.9 1_cp39 conda-forge
pywavelets 1.1.1 py39hce5d2b2_3 conda-forge
pyyaml 5.4.1 py39h3811e60_0 conda-forge
qt 5.12.9 hda022c4_4 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
scikit-image 0.18.1 py39hde0f152_0 conda-forge
scipy 1.6.3 py39hee8e79c_0 conda-forge
setuptools 49.6.0 py39hf3d152e_3 conda-forge
shapely 1.7.1 py39ha61afbd_5 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
sqlite 3.35.5 h74cdb3f_0 conda-forge
tifffile 2021.4.8 pyhd8ed1ab_0 conda-forge
tiledb 2.2.9 h91fcb0e_0 conda-forge
tk 8.6.10 h21135ba_1 conda-forge
toolz 0.11.1 py_0 conda-forge
tornado 6.1 py39h3811e60_1 conda-forge
tzcode 2021a h7f98852_1 conda-forge
tzdata 2021a he74cb21_0 conda-forge
urllib3 1.26.5 pyhd8ed1ab_0 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
xerces-c 3.2.3 h9d8b166_2 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zfp 0.5.5 h9c3ff4c_5 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.9 ha95c52a_0 conda-forge
Thanks for looking into this. The conda env was created ~1 week ago (details below). I retried running the with your minimum settings file. Processing gets further (I think because it’s using ReverseLUTs path now) but still craps out with a similar error.
-brian
> cat acolite_basic.cfg
output=/work/b/bbarnes4/processing/completed/E2021160160509.L2_basic/
inputfile=/work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
limit=25.25,-80.4,25.45,-80.1
l2w_parameters=Rrs_*,rhorc_*,chl_oc3,fai
> python /shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py --cli --settings /work/b/bbarnes4/processing/working3/E2021160160509/acolite_basic.cfg
Running generic ACOLITE processing - Generic Version Public Beta - April 2021
Run ID - 20210616_095807
/work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
/work/b/bbarnes4/processing/completed/E2021160160509.L2_basic//acolite_run_20210616_095807_l1r_settings.txt
Starting conversion of 1 scenes
Starting conversion of /work/b/bbarnes4/processing/working3/E2021160160509/S2B_MSIL1C_20210609T160509_N0300_R054_T17RNJ_20210609T204554.SAFE/
Importing metadata from L1C_T17RNJ_A022245_20210609T160548
Reading per pixel geometry
Computing per detector geometry
Detector 2
/shares/cms_optics/apps/acolite_20210114/acolite/acolite/sentinel2/l1_convert.py:359: RuntimeWarning: Mean of empty slice
ave_vza = np.nanmean(ave_vza, axis=2)
/shares/cms_optics/apps/acolite_20210114/acolite/acolite/sentinel2/l1_convert.py:360: RuntimeWarning: Mean of empty slice
ave_vaa = np.nanmean(ave_vaa, axis=2)
Detector 6
Detector 7
Detector 4
Detector 5
Detector 3
Wrote raa
Wrote vza
Wrote sza
Writing geolocation lon/lat
(2232, 3033)
Wrote lon
(2232, 3033)
Wrote lat
Converting bands
Converting bands: Wrote rhot_442 ((2232, 3033))
Converting bands: Wrote rhot_492 ((2232, 3033))
Converting bands: Wrote rhot_559 ((2232, 3033))
Converting bands: Wrote rhot_665 ((2232, 3033))
Converting bands: Wrote rhot_704 ((2232, 3033))
Converting bands: Wrote rhot_739 ((2232, 3033))
Converting bands: Wrote rhot_780 ((2232, 3033))
Converting bands: Wrote rhot_833 ((2232, 3033))
Converting bands: Wrote rhot_864 ((2232, 3033))
Converting bands: Wrote rhot_943 ((2232, 3033))
Converting bands: Wrote rhot_1377 ((2232, 3033))
Converting bands: Wrote rhot_1610 ((2232, 3033))
Converting bands: Wrote rhot_2186 ((2232, 3033))
Conversion took 16.9 seconds
Created /work/b/bbarnes4/processing/completed/E2021160160509.L2_basic//S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc
Running acolite for /work/b/bbarnes4/processing/completed/E2021160160509.L2_basic//S2B_MSI_2021_06_09_16_06_14_T17RNJ_L1R.nc
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
EARTHDATA user name and password required for download of https://oceandata.sci.gsfc.nasa.gov/ob/ ... 6h.hdf.bz2
Using DSF atmospheric correction
Processing with 24 tiles (4x6)
Writing lon
Wrote lon
Writing lat
Wrote lat
1 rhot_442
S2B_MSI/B1 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B1 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
2 rhot_492
S2B_MSI/B2 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B2 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
3 rhot_559
S2B_MSI/B3 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B3 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
4 rhot_665
S2B_MSI/B4 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B4 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
5 rhot_704
S2B_MSI/B5 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B5 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
6 rhot_739
S2B_MSI/B6 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B6 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
7 rhot_780
S2B_MSI/B7 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B7 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
8 rhot_833
S2B_MSI/B8 ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B8 ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
8A rhot_864
S2B_MSI/B8A ACOLITE-LUT-202102-MOD1 took 0.003s (RevLUT)
S2B_MSI/B8A ACOLITE-LUT-202102-MOD2 took 0.003s (RevLUT)
Choosing best fitting model: min_drmsd
Computing RMSD for model ACOLITE-LUT-202102-MOD1
Computing RMSD for model ACOLITE-LUT-202102-MOD2
use_revlut True
Wrote rhot_442
Computing surface reflectance 1 442 0.998
Interpolating tiles
Traceback (most recent call last):
File "/shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py", line 64, in <module>
launch_acolite()
File "/shares/cms_optics/apps/acolite_20210114/acolite/launch_acolite.py", line 58, in launch_acolite
ac.acolite.acolite_run(args.settings, inputfile=inputfile, output=output)
File "/shares/cms_optics/apps/acolite_20210114/acolite/acolite/acolite/acolite_run.py", line 108, in acolite_run
ret = ac.acolite.acolite_l2r(l1r, settings = setu, verbosity = verbosity)
File "/shares/cms_optics/apps/acolite_20210114/acolite/acolite/acolite/acolite_l2r.py", line 913, in acolite_l2r
cur_rhorc = (cur_rhorc - rorayl_cur) / (dtotr_cur*utotr_cur)
ValueError: operands could not be broadcast together with shapes (2232,3033) (4,6)
> ll -d /home/b/bbarnes4/.anaconda3/envs/ACOLITE_BBB/
drwxrwxr-x 24 bbarnes4 usfuser 4096 Jun 10 11:47 /home/b/bbarnes4/.anaconda3/envs/ACOLITE_BBB/
> conda list --name ACOLITE_BBB
# packages in environment at /home/b/bbarnes4/.anaconda3/envs/ACOLITE_BBB:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
alsa-lib 1.2.3 h516909a_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
astropy 4.2.1 py39hce5d2b2_0 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
boost-cpp 1.74.0 hc6e9bd1_3 conda-forge
brotli 1.0.9 h9c3ff4c_4 conda-forge
brotlipy 0.7.0 py39h3811e60_1001 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h7f98852_1 conda-forge
ca-certificates 2021.5.30 ha878542_0 conda-forge
cairo 1.16.0 h6cf1ce9_1008 conda-forge
cartopy 0.19.0.post1 py39h3b23250_0 conda-forge
certifi 2021.5.30 py39hf3d152e_0 conda-forge
cffi 1.14.5 py39he32792d_0 conda-forge
cfitsio 3.470 hb418390_7 conda-forge
cftime 1.5.0 py39hce5d2b2_0 conda-forge
chardet 4.0.0 py39hf3d152e_1 conda-forge
charls 2.2.0 h9c3ff4c_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
configobj 5.0.6 py_0 conda-forge
cryptography 3.4.7 py39hbca0aa6_0 conda-forge
curl 7.77.0 hea6ffbf_0 conda-forge
cycler 0.10.0 py_2 conda-forge
cytoolz 0.11.0 py39h3811e60_3 conda-forge
dask-core 2021.6.0 pyhd8ed1ab_0 conda-forge
dbus 1.13.6 h48d8840_2 conda-forge
decorator 5.0.9 pyhd8ed1ab_0 conda-forge
expat 2.4.1 h9c3ff4c_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
fsspec 2021.6.0 pyhd8ed1ab_0 conda-forge
gdal 3.2.1 py39h409cc32_7 conda-forge
geos 3.9.1 h9c3ff4c_2 conda-forge
geotiff 1.6.0 h2b14fbe_4 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glib 2.68.2 h9c3ff4c_2 conda-forge
glib-tools 2.68.2 h9c3ff4c_2 conda-forge
gst-plugins-base 1.18.4 hf529b03_2 conda-forge
gstreamer 1.18.4 h76c114f_2 conda-forge
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
icu 68.1 h58526e2_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagecodecs 2021.3.31 py39h559889c_0 conda-forge
imageio 2.9.0 py_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
kealib 1.4.14 hcc255d8_2 conda-forge
kiwisolver 1.3.1 py39h1a9c180_1 conda-forge
krb5 1.19.1 hcc1bbae_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge
lerc 2.2.1 h9c3ff4c_0 conda-forge
libaec 1.0.4 h9c3ff4c_1 conda-forge
libblas 3.9.0 9_openblas conda-forge
libcblas 3.9.0 9_openblas conda-forge
libclang 11.1.0 default_ha53f305_1 conda-forge
libcurl 7.77.0 h2574ce0_0 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.7 h7f98852_5 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libffi 3.3 h58526e2_2 conda-forge
libgcc-ng 9.3.0 h2828fa1_19 conda-forge
libgdal 3.2.1 h38ff51b_7 conda-forge
libgfortran-ng 9.3.0 hff62375_19 conda-forge
libgfortran5 9.3.0 hff62375_19 conda-forge
libglib 2.68.2 h3e27bee_2 conda-forge
libgomp 9.3.0 h2828fa1_19 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h238a007_1013 conda-forge
liblapack 3.9.0 9_openblas conda-forge
libllvm11 11.1.0 hf817b99_2 conda-forge
libnetcdf 4.7.4 nompi_h56d31a8_107 conda-forge
libnghttp2 1.43.0 h812cca2_0 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.15 pthreads_h8fe5266_1 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 13.3 hd57d9b9_0 conda-forge
librttopo 1.1.0 h1185371_6 conda-forge
libspatialite 5.0.1 he52d314_3 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 h6de172a_19 conda-forge
libtiff 4.2.0 hbd63e13_2 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxcb 1.13 h7f98852_1003 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.12 h72842e0_0 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
matplotlib 3.4.2 py39hf3d152e_0 conda-forge
matplotlib-base 3.4.2 py39h2fa2bec_0 conda-forge
mysql-common 8.0.23 ha770c72_2 conda-forge
mysql-libs 8.0.23 h935591d_2 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
netcdf4 1.5.6 nompi_py39h36800e2_102 conda-forge
networkx 2.5 py_0 conda-forge
nspr 4.30 h9c3ff4c_0 conda-forge
nss 3.66 hb5efdd6_0 conda-forge
numpy 1.20.3 py39hdbf815f_1 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
packaging 20.9 pyh44b312d_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pillow 8.2.0 py39hf95b381_1 conda-forge
pip 21.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.3.0 pyhd8ed1ab_0 conda-forge
poppler 0.89.0 h2de54a5_5 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 13.3 h2510834_0 conda-forge
proj 7.2.0 h277dcde_2 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyerfa 1.7.3 py39hce5d2b2_1 conda-forge
pyhdf 0.10.3 py39hb0d3d0c_0 conda-forge
pykdtree 1.3.4 py39hce5d2b2_0 conda-forge
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyproj 3.0.1 py39h50a64a9_0 conda-forge
pyqt 5.12.3 py39hf3d152e_7 conda-forge
pyqt-impl 5.12.3 py39h0fcd23e_7 conda-forge
pyqt5-sip 4.19.18 py39he80948d_7 conda-forge
pyqtchart 5.12 py39h0fcd23e_7 conda-forge
pyqtwebengine 5.12.1 py39h0fcd23e_7 conda-forge
pyresample 1.20.0 py39hde0f152_0 conda-forge
pyshp 2.1.3 pyh44b312d_0 conda-forge
pysocks 1.7.1 py39hf3d152e_3 conda-forge
python 3.9.4 hffdb5ce_0_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.9 1_cp39 conda-forge
pywavelets 1.1.1 py39hce5d2b2_3 conda-forge
pyyaml 5.4.1 py39h3811e60_0 conda-forge
qt 5.12.9 hda022c4_4 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
scikit-image 0.18.1 py39hde0f152_0 conda-forge
scipy 1.6.3 py39hee8e79c_0 conda-forge
setuptools 49.6.0 py39hf3d152e_3 conda-forge
shapely 1.7.1 py39ha61afbd_5 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
sqlite 3.35.5 h74cdb3f_0 conda-forge
tifffile 2021.4.8 pyhd8ed1ab_0 conda-forge
tiledb 2.2.9 h91fcb0e_0 conda-forge
tk 8.6.10 h21135ba_1 conda-forge
toolz 0.11.1 py_0 conda-forge
tornado 6.1 py39h3811e60_1 conda-forge
tzcode 2021a h7f98852_1 conda-forge
tzdata 2021a he74cb21_0 conda-forge
urllib3 1.26.5 pyhd8ed1ab_0 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
xerces-c 3.2.3 h9d8b166_2 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zfp 0.5.5 h9c3ff4c_5 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.9 ha95c52a_0 conda-forge
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Hi Brian
The original error must be caused by a setting in your first settings file. I'll have to have a closer look!
The "ValueError: operands could not be broadcast together with shapes (2232,3033) (4,6)" should be fixed with yesterday's commits, I forgot to interpolate the tiled result (4,6) into the full image dimensions (2232,3033)...
I also did another commit this morning to switch Rayleigh correction to using the results from aerosol optical depth = 0.001, which will allow you to switch from using Rayleigh correction with or without the air-water interface contribution (sky_correction=True or False, "sky" includes the sun glint image on the surface, so is a bit of poor name). The previous version had only the option with pure Rayleigh reflectance, i.e. for a black surface.
If you are only interested in Rayleigh correction (or actually for your rather small ROI), I would probably recommend the use of dsf_path_reflectance=fixed. (Although it can be quite a bit slower due to per pixel geometry interpolation, especially when computing both rhos and rhorc. How much processing time you can live with depends a bit on how many scenes you are interested in.) There was a similar topic pretty much at the same time as yours: viewtopic.php?f=4&t=267
Hope this helps!
Quinten
The original error must be caused by a setting in your first settings file. I'll have to have a closer look!
The "ValueError: operands could not be broadcast together with shapes (2232,3033) (4,6)" should be fixed with yesterday's commits, I forgot to interpolate the tiled result (4,6) into the full image dimensions (2232,3033)...
I also did another commit this morning to switch Rayleigh correction to using the results from aerosol optical depth = 0.001, which will allow you to switch from using Rayleigh correction with or without the air-water interface contribution (sky_correction=True or False, "sky" includes the sun glint image on the surface, so is a bit of poor name). The previous version had only the option with pure Rayleigh reflectance, i.e. for a black surface.
If you are only interested in Rayleigh correction (or actually for your rather small ROI), I would probably recommend the use of dsf_path_reflectance=fixed. (Although it can be quite a bit slower due to per pixel geometry interpolation, especially when computing both rhos and rhorc. How much processing time you can live with depends a bit on how many scenes you are interested in.) There was a similar topic pretty much at the same time as yours: viewtopic.php?f=4&t=267
Hope this helps!
Quinten
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Thanks again, Quinten.
Processing this file works with dsf_path_reflectance=fixed, both on my previous version and the latest git pull (today). Not sure what my issue is with dsf_path_reflectance=tiled (I tried playing with a ton of options in the .cfg, but always got the same error). Anyway, I don't mind using the fixed option...
-brian
Processing this file works with dsf_path_reflectance=fixed, both on my previous version and the latest git pull (today). Not sure what my issue is with dsf_path_reflectance=tiled (I tried playing with a ton of options in the .cfg, but always got the same error). Anyway, I don't mind using the fixed option...
-brian
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Thanks for the update Brian. Do you still have the issue with tiled processing with today's pull? These lines should fix the shape mismatch issue in the tiled processing.
Quinten
Quinten
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Quinten-
Sorry for the delay in getting back to you. Yes, I'm still having the same issue after a pull today (6/23):
>/shares/cms_optics/etc/shared_conda/ACOLITE/bin/python3 /shares/cms_optics/apps/acolite/acolite/launch_acolite.py --cli --settings /work/b/bbarnes4/processing/working2/L2021149015042/acolite.cfg
Running generic ACOLITE processing - Generic Version Public Beta - April 2021
Run ID - 20210623_143049
/work/b/bbarnes4/processing/working2/L2021149015042/
/work/b/bbarnes4/processing/completed2/L2021149015042.L2/acolite_run_20210623_143049_l1r_settings.txt
Starting conversion of 1 scenes
Starting conversion of /work/b/bbarnes4/processing/working2/L2021149015042/
Importing metadata from /work/b/bbarnes4/processing/working2/L2021149015042/
/work/b/bbarnes4/processing/completed2/L2021149015042.L2/L8_OLI_2021_05_29_15_49_45_015042_L1R.nc
Reading per pixel geometry
Wrote raa
Wrote vza
Wrote sza
Writing geolocation lon/lat
Wrote lon
Wrote lat
Converting bands
Converting bands: Wrote rhot_443 ((744, 1012))
Converting bands: Wrote rhot_483 ((744, 1012))
Converting bands: Wrote rhot_561 ((744, 1012))
Converting bands: Wrote rhot_655 ((744, 1012))
Converting bands: Wrote rhot_865 ((744, 1012))
Converting bands: Wrote rhot_1609 ((744, 1012))
Converting bands: Wrote rhot_2201 ((744, 1012))
Converting bands: Wrote rhot_592 to separate L1R_pan
Converting bands: Wrote rhot_592 ((744, 1012))
Converting bands: Wrote rhot_1373 ((744, 1012))
Converting bands: Wrote bt10
Converting bands: Wrote bt11
Conversion took 3.9 seconds
Created /work/b/bbarnes4/processing/completed2/L2021149015042.L2/L8_OLI_2021_05_29_15_49_45_015042_L1R.nc
Running acolite for /work/b/bbarnes4/processing/completed2/L2021149015042.L2/L8_OLI_2021_05_29_15_49_45_015042_L1R.nc
L8_OLI_2021_05_29_15_49_45_015042_L1R
Using DSF atmospheric correction
sza
vza
raa
Processing with 24 tiles (4x6)
Writing lon
Wrote lon
Writing lat
Wrote lat
Loading LUTs
Loading LUTs took 16.7 s
1 rhot_443
Traceback (most recent call last):
File "/shares/cms_optics/apps/acolite/acolite/launch_acolite.py", line 64, in <module>
launch_acolite()
File "/shares/cms_optics/apps/acolite/acolite/launch_acolite.py", line 58, in launch_acolite
ac.acolite.acolite_run(args.settings, inputfile=inputfile, output=output)
File "/shares/cms_optics/apps/acolite_20210114/git_20210623/acolite/acolite/acolite/acolite_run.py", line 108, in acolite_run
ret = ac.acolite.acolite_l2r(l1r, settings = setu, verbosity = verbosity)
File "/shares/cms_optics/apps/acolite_20210114/git_20210623/acolite/acolite/acolite/acolite_l2r.py", line 457, in acolite_l2r
tmp = lutdw[lut]['rgi']((gem.data_mem['pressure'+gk],
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/scipy/interpolate/interpolate.py", line 2496, in __call__
xi = _ndim_coords_from_arrays(xi, ndim=ndim)
File "interpnd.pyx", line 157, in scipy.interpolate.interpnd._ndim_coords_from_arrays
File "interpnd.pyx", line 168, in scipy.interpolate.interpnd._ndim_coords_from_arrays
File "<__array_function__ internals>", line 5, in broadcast_arrays
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/numpy/lib/stride_tricks.py", line 538, in broadcast_arrays
shape = _broadcast_shape(*args)
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/numpy/lib/stride_tricks.py", line 420, in _broadcast_shape
b = np.broadcast(*args[:32])
ValueError: shape mismatch: objects cannot be broadcast to a single shape
Sorry for the delay in getting back to you. Yes, I'm still having the same issue after a pull today (6/23):
>/shares/cms_optics/etc/shared_conda/ACOLITE/bin/python3 /shares/cms_optics/apps/acolite/acolite/launch_acolite.py --cli --settings /work/b/bbarnes4/processing/working2/L2021149015042/acolite.cfg
Running generic ACOLITE processing - Generic Version Public Beta - April 2021
Run ID - 20210623_143049
/work/b/bbarnes4/processing/working2/L2021149015042/
/work/b/bbarnes4/processing/completed2/L2021149015042.L2/acolite_run_20210623_143049_l1r_settings.txt
Starting conversion of 1 scenes
Starting conversion of /work/b/bbarnes4/processing/working2/L2021149015042/
Importing metadata from /work/b/bbarnes4/processing/working2/L2021149015042/
/work/b/bbarnes4/processing/completed2/L2021149015042.L2/L8_OLI_2021_05_29_15_49_45_015042_L1R.nc
Reading per pixel geometry
Wrote raa
Wrote vza
Wrote sza
Writing geolocation lon/lat
Wrote lon
Wrote lat
Converting bands
Converting bands: Wrote rhot_443 ((744, 1012))
Converting bands: Wrote rhot_483 ((744, 1012))
Converting bands: Wrote rhot_561 ((744, 1012))
Converting bands: Wrote rhot_655 ((744, 1012))
Converting bands: Wrote rhot_865 ((744, 1012))
Converting bands: Wrote rhot_1609 ((744, 1012))
Converting bands: Wrote rhot_2201 ((744, 1012))
Converting bands: Wrote rhot_592 to separate L1R_pan
Converting bands: Wrote rhot_592 ((744, 1012))
Converting bands: Wrote rhot_1373 ((744, 1012))
Converting bands: Wrote bt10
Converting bands: Wrote bt11
Conversion took 3.9 seconds
Created /work/b/bbarnes4/processing/completed2/L2021149015042.L2/L8_OLI_2021_05_29_15_49_45_015042_L1R.nc
Running acolite for /work/b/bbarnes4/processing/completed2/L2021149015042.L2/L8_OLI_2021_05_29_15_49_45_015042_L1R.nc
L8_OLI_2021_05_29_15_49_45_015042_L1R
Using DSF atmospheric correction
sza
vza
raa
Processing with 24 tiles (4x6)
Writing lon
Wrote lon
Writing lat
Wrote lat
Loading LUTs
Loading LUTs took 16.7 s
1 rhot_443
Traceback (most recent call last):
File "/shares/cms_optics/apps/acolite/acolite/launch_acolite.py", line 64, in <module>
launch_acolite()
File "/shares/cms_optics/apps/acolite/acolite/launch_acolite.py", line 58, in launch_acolite
ac.acolite.acolite_run(args.settings, inputfile=inputfile, output=output)
File "/shares/cms_optics/apps/acolite_20210114/git_20210623/acolite/acolite/acolite/acolite_run.py", line 108, in acolite_run
ret = ac.acolite.acolite_l2r(l1r, settings = setu, verbosity = verbosity)
File "/shares/cms_optics/apps/acolite_20210114/git_20210623/acolite/acolite/acolite/acolite_l2r.py", line 457, in acolite_l2r
tmp = lutdw[lut]['rgi']((gem.data_mem['pressure'+gk],
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/scipy/interpolate/interpolate.py", line 2496, in __call__
xi = _ndim_coords_from_arrays(xi, ndim=ndim)
File "interpnd.pyx", line 157, in scipy.interpolate.interpnd._ndim_coords_from_arrays
File "interpnd.pyx", line 168, in scipy.interpolate.interpnd._ndim_coords_from_arrays
File "<__array_function__ internals>", line 5, in broadcast_arrays
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/numpy/lib/stride_tricks.py", line 538, in broadcast_arrays
shape = _broadcast_shape(*args)
File "/shares/cms_optics/etc/shared_conda/ACOLITE/lib/python3.9/site-packages/numpy/lib/stride_tricks.py", line 420, in _broadcast_shape
b = np.broadcast(*args[:32])
ValueError: shape mismatch: objects cannot be broadcast to a single shape
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Hi Brian
Processing worked here with a minimal settings file, so it is probably one of your non-default settings that is causing the problem. The last commit just now fixed a problem when resolved_geometry=False for input bundles which have in fact resolved geometry.
Note that by default this is now resolved_geometry=True for S2 and Landsat (see config/defaults/). As of Collection2 Landsat has per pixel geometry included in the L1 files!
I haven't looked at all your settings yet, but I think this was the main problem. Let me know how it goes!
Quinten
Processing worked here with a minimal settings file, so it is probably one of your non-default settings that is causing the problem. The last commit just now fixed a problem when resolved_geometry=False for input bundles which have in fact resolved geometry.
Note that by default this is now resolved_geometry=True for S2 and Landsat (see config/defaults/). As of Collection2 Landsat has per pixel geometry included in the L1 files!
I haven't looked at all your settings yet, but I think this was the main problem. Let me know how it goes!
Quinten
-
IsabelBrand
- Posts: 45
- Joined: Wed Apr 19, 2017 9:37 am
Re: ValueError: shape mismatch: objects cannot be broadcast to a single shape
Hi Quinten,
I downloaded the new version of acolite this morning, and I am having the same problem as Brian regarding shape mismatch. I've tried your suggestions with resolved_geometry (both True and False), but I get this error just after the line below...
Loading LUTs
Loading LUTs took 0.6 s
1 rhot_486
By the way, I input L5 for this exercise.
Isabel
I downloaded the new version of acolite this morning, and I am having the same problem as Brian regarding shape mismatch. I've tried your suggestions with resolved_geometry (both True and False), but I get this error just after the line below...
Loading LUTs
Loading LUTs took 0.6 s
1 rhot_486
By the way, I input L5 for this exercise.
Isabel