In ac.ac.ancillary.interp_met, any .bz2 files are unzipped (L31), then the original .bz2 is deleted (L68).
However, if the same ancillary file is needed in a subsequent acolite run, when ac.ac.ancillary.download queries ac.ac.ancillary.list_files, the filenames with .bz2 are returned (e.g., N202116018_MET_NCEPR2_6h.hdf.bz2). So, ac.ac.ancillary.download will look for that filename (L48), not find it, and thus download (L56) again.
Is it possible to query met_path for both zipped and unzipped versions of the ancillary files? Thus, L48 in download.py would change from :
if os.path.exists(local_file) & (not override):
to something like:
local_unzipped=local_file.split('.bz2')[0] if '.bz2' in local_file else local_file
if ((os.path.exists(local_file)) | (os.path.exists(local_unzipped))) & (not override):
Zipped EarthData MET files are re-downloaded in subsequent runs
Re: Zipped EarthData MET files are re-downloaded in subsequent runs
Hi Brian
In ac.ac.ancillary.interp_met, the original .bz2 file should not be deleted, but please confirm if you find otherwise. If a zipped file is passed, the lines at 28--32 should unzip the file, and change the reference from the original file to the bz2 file, and create a new "file_zipped" variable. The deletion of "file" at line 68 should be the unzipped file.
I usually keep the .bz2 files and unzip when needed, but if you get/unzip the bz2 files through another process ACOLITE may indeed interfere with those files, i.e. redownload .bz2 and delete the .hdf file. I have now updated ac.ac.ancillary_download.py to test also for the presence of a zipped or unzipped file, which should make this work as intended. The update should only download the .bz2 if the .bz2 or .hdf is missing, and the interp function should leave the .hdf file alone if it did not unzip it itself.
Let me know if this fix works for you!
Quinten
In ac.ac.ancillary.interp_met, the original .bz2 file should not be deleted, but please confirm if you find otherwise. If a zipped file is passed, the lines at 28--32 should unzip the file, and change the reference from the original file to the bz2 file, and create a new "file_zipped" variable. The deletion of "file" at line 68 should be the unzipped file.
I usually keep the .bz2 files and unzip when needed, but if you get/unzip the bz2 files through another process ACOLITE may indeed interfere with those files, i.e. redownload .bz2 and delete the .hdf file. I have now updated ac.ac.ancillary_download.py to test also for the presence of a zipped or unzipped file, which should make this work as intended. The update should only download the .bz2 if the .bz2 or .hdf is missing, and the interp function should leave the .hdf file alone if it did not unzip it itself.
Let me know if this fix works for you!
Quinten
Re: Zipped EarthData MET files are re-downloaded in subsequent runs
Gotcha. Thanks so much for the update on this. You are correct that the .bz2 file wasn't deleted - I was having crosstalk with other systems that use the MET data (which are resolved with the new commit).
Thanks again
-brian
Thanks again
-brian