acolite download ancillary data, but does not use them

[jcs1973]

Hello

We are using acolite, we have activated the ancillary data download for atmospheric correction. The ancillary data were correctly downloaded (see example):

/home/images/acolite_github/acolite/data/MET/2020/051
-rw-r--r-- 1 seasnet seasnet 854267 mar 4 12:34 N202005100_MET_NCEPR2_6h.hdf.bz2
-rw-r--r-- 1 seasnet seasnet 201285 mar 4 12:35 N202005100_O3_AURAOMI_24h.hdf
-rw-r--r-- 1 seasnet seasnet 850128 mar 4 12:35 N202005106_MET_NCEPR2_6h.hdf.bz2
-rw-r--r-- 1 seasnet seasnet 854355 mar 4 12:35 N202005112_MET_NCEPR2_6h.hdf.bz2
-rw-r--r-- 1 seasnet seasnet 853920 mar 4 12:35 N202005118_MET_NCEPR2_6h.hdf.bz2
-rw-r--r-- 1 seasnet seasnet 162082 mar 4 12:34 S20200510005123_TOAST.OZONE
/home/images/acolite_github/acolite/data/MET/2020/052
-rw-r--r-- 1 seasnet seasnet 853915 mar 4 12:35 N202005200_MET_NCEPR2_6h.hdf.bz2

However in the processing log file we get the following output:

2020-03-04 10:59:33: Could not retrieve ancillary data, proceeding with default values.
2020-03-04 10:59:33: No ancillary pressure found: using default.
2020-03-04 10:59:33: No ancillary ozone found: using default 0.3.
2020-03-04 10:59:33: No ancillary ozone found: using default 1.5.


The size of the files is correct, the interactive visualization of ancillary files using SeaDAS is OK. In some cases, the problem was related to an incorrect files download, after removing them and relaunching the processing, the correct ancillary files were get, however the output persist.

Please find enclosed the acolite settings and log file for an example. We are using linux acolite python code just downloaded from github. The same happens with linux binary acolite 20190326.


Best regards
Josep

[quinten]

Hi Josep

What are the sizes of those files?

Quinten

[jcs1973]

Hello Quinten

The size of these ancillary data is:

835K N202005100_MET_NCEPR2_6h.hdf.bz2
197K N202005100_O3_AURAOMI_24h.hdf
831K N202005106_MET_NCEPR2_6h.hdf.bz2
835K N202005112_MET_NCEPR2_6h.hdf.bz2
834K N202005118_MET_NCEPR2_6h.hdf.bz2
159K S20200510005123_TOAST.OZONE
834K N202005200_MET_NCEPR2_6h.hdf.bz2

The N202005*_MET_NCEPR2_6h.hdf uncompressed files has their corresponding 1,6 Mb file size. I have checked the size of the files because we also process MODIS and VIIRS data using SeaDAS and in some isolated cases we experienced problems with the ancillary data download. In those cases to recover the processing, the incomplete data remove is also required. All the ancillary data in this post, at least apperantly, are correctly showed in SeaDAS and SNAP.

Josep

[quinten]

Hi Josep

There is indeed a problem with the binary distribution and using the ancillary data.

For the code from GitHub I believe it should work when the MET files are present. (I just tested for your scene with a fresh clone.) I'm working on downloading the files which currently requires the specification of a EarthData login and password.

Quinten

[quinten]

Hi Josep

Download should now work if you put your EarthData credentials in the config/acolite_config.txt file:

Code: Select all

## EarthData
EARTHDATA_u=
EARTHDATA_p=

Alternatively you can export them to environment variables which should be accessed by ACOLITE.

Quinten

[jcs1973]

Hello Quinten

I have added
## EarthData
EARTHDATA_u=
EARTHDATA_p=
to config/acolite_config.txt file

Although that is not the problem, the ancillary data download was already OK in my system. The problem is why acolite log is showing/doing this?:

2020-03-11 16:38:31: Processing 1 scene...
2020-03-11 16:38:31: Processing scene 1 of 1...
2020-03-11 16:38:31: 2020-03-11 16:38:31 - Processing L1C_T27RYL_A014878_20200111T115213
2020-03-11 16:38:59: Could not retrieve ancillary data, proceeding with default values.
2020-03-11 16:39:00: No ancillary pressure found: using default.
2020-03-11 16:39:00: No ancillary ozone found: using default 0.3.
2020-03-11 16:39:00: No ancillary ozone found: using default 1.5.

The ancillary are properly retrieved in the acolite folder data/MET/YYYY/JJJ/

Josep

[quinten]

Hi Josep

Since you are using the code from GitHub, could you edit the acolite_ac.py file, and comment out these lines:

Code: Select all

            try:
                pc_anc = pp.ac.ancillary.ancillary_get(pc_date, pc_lon, pc_lat, ftime=pc_time, kind='nearest', verbosity=verbosity)
            except:
                pc_anc = {}
                print('Could not retrieve ancillary data, proceeding with default values.')

and add these ones below:

Code: Select all

            #try:
            #    pc_anc = pp.ac.ancillary.ancillary_get(pc_date, pc_lon, pc_lat, ftime=pc_time, kind='nearest', verbosity=verbosity)
            #except:
            #    pc_anc = {}
            #    print('Could not retrieve ancillary data, proceeding with default values.')
            
            pc_anc = pp.ac.ancillary.ancillary_get(pc_date, pc_lon, pc_lat, ftime=pc_time, kind='nearest', verbosity=verbosity)
            print(pc_anc)

and post the output here? The try/except construct hides where exactly the processing of the ancillary data is failing.

Thanks!

Quinten

[jcs1973]

Dear Quinten

I did the changes and I looked and the output, the problem was related to the lack of pyhdf python module on my system. So acolite could not read the hdf ancillary files.

I have installed the pyhdf module and reverted the changes that you suggested to identify the problem, and now the messages of missing ancillary files have disappeared.

Thank you very much!!
Josep

[quinten]

Thanks Josep!

Quinten

[arunocean]

Dear Quinten

I did the changes and I looked and the output, the problem was related to the lack of pyhdf python module on my system. So acolite could not read the hdf ancillary files.

I have installed the pyhdf module and reverted the changes that you suggested to identify the problem, and now the messages of missing ancillary files have disappeared.

Thank you very much!!
Josep

Hi Joseph,

I'm also facing a similar issue that you had faced. Is it really due to the lack of pyhdf module? As we don't have access to the internet, I have downloaded the ancillary datasets and kept under MET folder. Then, I ran the ACOLITE through CLI. In that case, I'm getting the same error that you got. How to know whether ACOLITE is reading the data or not. As per your experience, if pyhdf module can solve this, Can you please guide me on how to install pyhdf, please. I'm new to python.

regards
Arun