speeding up sentinel-2 processing

[ags-tolson]

I'm trying to perform acolite processing on single tiles from sentinel-2; these are the old-style (pre-dec-2016) assets that contain many tiles. I saw the 'limit' parameter that lets you specify a region of interest in the docs, but I can't seem to find any evidence that it's actually processing any faster. For instance, previously a new-style asset (with a single tile) can be finished in less than an hour, but so far old-style assets have needed four to eight hours. Moreover I get this message in the output:

Code: Select all

No region given, using default UTM zone: 16

Here's the shell command I'm using to start it:

Code: Select all

$ idl -IDL_CPU_TPOOL_NTHREADS 1 -rt=acolite.sav -args settings=/path/to/settings.cfg run=1 output=/path/to/output
image=/path/to/S2A_OPER_PRD_MSIL1C_PDMC_20160829T180030_R040_V20150804T163126_20150804T163120.SAFE
limit=41.4077261895,-84.6450457662,42.4293608719,-83.2769665673

Do I have to specify 'limit' in the config file? Right now it's on the command line and not in acolite.cfg.

[quinten]

Yes, the limit parameter should go in the config file. Depending on the size of your ROI it should be quite a bit faster!

You could also disable the generation of PNG maps to speed things up. If you are using spatial resampling of the SWIR bands that will also increase processing time. For a small ROI you could fix the aerosol reflectance to further speed things up.

I see you are limiting the number of threads in IDL as well, probably because you are running your own parallel processing? On a few year old desktop (quad core) I can process a single tile in about 15-20 minutes with the standard IDL parallelization ("thread pool").

[ags-tolson]

Thanks so much, limit= in the config file did the trick; single-threaded processing went down to about an hour on a single CPU of our small cluster (I can't remember if it was rhow alone or rhow + other products).

You're right that we're doing our own parallelization; our setup, which is hopefully temporary, is to submit a single job specification to PBS/Torque, and then the torque job spawns acolite processing runs with GNU parallel, generally one per scene. We use our own GIS management software, gips, to organize processing and files storage:

https://github.com/Applied-GeoSolutions/gips/