Recent problems downloading ancillary data

Technical questions about ACOLITE
fmaciel
Posts: 10
Joined: Thu Dec 13, 2018 8:16 pm

Recent problems downloading ancillary data

Post by fmaciel »

Hi,
I have been successfully using Acolite for a while, and since the new Python version came out I have never had a problem with the ancillary data. But recently it began to fail downloading ancillary data files:
"2019-09-22 23:38:49: Downloading file S20182630026323_TOAST.OZONE failed
2019-09-22 23:41:00: Downloading file N201826300_MET_NCEPR2_6h.hdf.bz2 failed
2019-09-22 23:43:11: Downloading file N201826306_MET_NCEPR2_6h.hdf.bz2 failed
2019-09-22 23:45:22: Downloading file N201826312_MET_NCEPR2_6h.hdf.bz2 failed
2019-09-22 23:47:33: Downloading file N201826318_MET_NCEPR2_6h.hdf.bz2 failed
2019-09-22 23:49:44: Downloading file N201826400_MET_NCEPR2_6h.hdf.bz2 failed
2019-09-22 23:51:55: Downloading file N201826300_O3_AURAOMI_24h.hdf failed"

It happens to me in both Windows and Linux versions. At first I thought it was the University proxy, but when I try to run it at home I get a different message but same result:
"Could not retrieve ancillary data, proceeding with default values."

I am wondering if this is happening to anyone else and do you have any idea what could be causing it? Until May (at least) it worked perfectly.

Thank you!
Fernanda
quinten
Posts: 991
Joined: Tue Mar 03, 2015 8:13 am

Re: Recent problems downloading ancillary data

Post by quinten »

Hi Fernanda

Thanks for your question! I can still get the ancillary data (using the Python code from GitHub) for recent scenes. Maybe it is a problem specific to the binary version, or perhaps the date you are trying to process data for.

Are you using the binary distribution? Could you share the scene identifier and ROI you are processing?

Quinten
fmaciel
Posts: 10
Joined: Thu Dec 13, 2018 8:16 pm

Re: Recent problems downloading ancillary data

Post by fmaciel »

Hello,
thanks for the response! I am using the binary distribution, here is a list of the images I was trying to process when I had the issue:
S2A_MSIL1C_20190918T134211_N0208_R124_T21HWB_20190918T201521.SAFE,
S2A_MSIL1C_20190620T134211_N0207_R124_T21HWB_20190620T170154.SAFE,
S2A_MSIL1C_20191008T134211_N0208_R124_T21HWB_20191008T152051.SAFE,
S2A_MSIL1C_20191018T134211_N0208_R124_T21HWB_20191018T165844.SAFE

They are all relatively recent scenes, and the ROI that I specify in the configuration file is:
limit=-35,-56.80,-34.73,-56.30

Thank you!
quinten
Posts: 991
Joined: Tue Mar 03, 2015 8:13 am

Re: Recent problems downloading ancillary data

Post by quinten »

Hi Fernanda

I am able to get the ancillary data for all those scenes, in the GitHub version of the code at least. Did you try removing the MET directory? Perhaps the file download was incomplete and the error comes from trying to open these damaged files.

Quinten
fmaciel
Posts: 10
Joined: Thu Dec 13, 2018 8:16 pm

Re: Recent problems downloading ancillary data

Post by fmaciel »

Thank you Quinten,

we found that apparently there is a problem of our university connection with the ocean data website, we are trying to solve that. Meanwhile, we were able to download the MET data using a different connection and independently from Acolite, but when we run Acolite it does not use this data.
We did the test for the image of February 10th 2019
We downloaded (manually) the following files (and we copied them in the MET directory in .\MET\2019\041):
N201904100_MET_NCEPR2_6h.hdf.bz2
N201904100_O3_AURAOMI_24h.hdf
N201904106_MET_NCEPR2_6h.hdf.bz2
N201904112_MET_NCEPR2_6h.hdf.bz2
N201904118_MET_NCEPR2_6h.hdf.bz2
S20190410004123_TOAST.OZONE

However, we get the following messages when running Acolite:
Processing 1 scene...
Processing scene 1 of 1...
2019-11-27 12:53:29 - Processing L1C_T21HVB_A018997_20190210T134207
Could not retrieve ancillary data, proceeding with default values.
No ancillary pressure found: using default.
No ancillary ozone found: using default 0.3.
No ancillary ozone found: using default 1.5.

It is not fail downloading the data, but it also does not read the ancillary data from the MET folder...are we missing any file?

Sorry to keep bothering you with this problem, I appreciate any help.

Thank you,
Fernanda
quinten
Posts: 991
Joined: Tue Mar 03, 2015 8:13 am

Re: Recent problems downloading ancillary data

Post by quinten »

Hi Fernanda

I hope I can look into more detail soon. Maybe it doesn't work with the manually downloaded files since you are missing the midnight file (201904200) for the next day.

Quinten
fmaciel
Posts: 10
Joined: Thu Dec 13, 2018 8:16 pm

Re: Recent problems downloading ancillary data

Post by fmaciel »

Hi Quinten,

a few updates regarding this issue: we solved the problem of our university connection with the ocean data website. Now I am able to correct images and download ancillary data using the 20181210.0 (binary) version of Acolite. However, with the latest version (20190326.0), the ancillary data gets downloaded into the MET file, but Acolite is not able to use it, it gives the error: "Could not retrieve ancillary data, proceeding with default values.", even when it downloaded the data. I tried removing the MET directory and I also tried the GitHub version (that I've recently downloaded), getting the same error.
I see only very small differences between the correction with the ancillary data and with the default values, but I still prefer to use the MET data. So for now I will keep using the 20181210.0 version. Could it be a bug in the latest version? Or nobody else has this problem?

Thank you for your help,
Fernanda
fmaciel
Posts: 10
Joined: Thu Dec 13, 2018 8:16 pm

Re: Recent problems downloading ancillary data

Post by fmaciel »

Could it be that there is a missing library in the latest binary version (20190326.0)? pyhdf?
quinten
Posts: 991
Joined: Tue Mar 03, 2015 8:13 am

Re: Recent problems downloading ancillary data

Post by quinten »

Hi Fernanda

What are the file sizes you are getting for the MET files downloaded by ACOLITE?

I noticed some recent issues with the Single Sign On for the EarthData.

Quinten
fmaciel
Posts: 10
Joined: Thu Dec 13, 2018 8:16 pm

Re: Recent problems downloading ancillary data

Post by fmaciel »

Hi Quinten,

I continue having this problem with the ancillary data, regardless of the Acolite version I use, it downloads the data but it is not able to use it. And as you suggested I've checked the MET file sizes and they have 11 KB. By looking at older images that were properly corrected the MET files weight between 150 and 830 KB depending on the file. Is this a problem with the EarthData website?

Thank you,
Fernanda
Post Reply