Free Data Access on The Cloud

Public datasets are now available to access via STScI’s public s3 bucket without the need for an AWS account. Missions that are currently available on the cloud are Hubble, Kepler, and TESS. The example below will demonstrate how public data for these missions can be accessed anonymously without needing AWS credentials.

>>> from astroquery.mast import Observations

>>> # Enable s3 public data anonymous access. The default provider is `AWS`, but we will write it in manually for this example:
>>> Observations.enable_cloud_dataset(provider='AWS')
INFO: Using the S3 STScI public dataset [astroquery.mast.core]

>>> # Getting the cloud URIs
>>> observations = Observations.query_criteria(obs_collection='HLSP',
                                               instrument_name='STIS/FUV-MAMA',
                                               proposal_id=['15629'],
                                               dataRights='PUBLIC')
>>> products = Observations.get_product_list(observations)
>>> filtered = Observations.filter_products(products,
                                            productType='SCIENCE')

Once the users have their desired products, these data can be downloaded from the cloud using download_products. To request for a cloud download, the cloud_only parameter must be set to True. When set to True, all data products that are not available in the cloud will be downloaded from the MAST on-premise database. If Observations.enable_cloud_dataset() is not called prior to this point, the cloud_only parameter is ignored.

>>> Observations.download_products(filtered, cloud_only=True)
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/av39a/dr4/hlsp_ullyses_hst_stis_av39a_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_av39a_e140m/hlsp_ullyses_hst_stis_av39a_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/av454/dr4/hlsp_ullyses_hst_stis_av454_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_av454_e140m/hlsp_ullyses_hst_stis_av454_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/av61/dr4/hlsp_ullyses_hst_stis_av61_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_av61_e140m/hlsp_ullyses_hst_stis_av61_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/av81/dr4/hlsp_ullyses_hst_stis_av81_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_av81_e140m/hlsp_ullyses_hst_stis_av81_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/vfts180/dr4/hlsp_ullyses_hst_stis_vfts180_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_vfts180_e140m/hlsp_ullyses_hst_stis_vfts180_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/vfts440/dr4/hlsp_ullyses_hst_stis_vfts440_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_vfts440_e140m/hlsp_ullyses_hst_stis_vfts440_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLSP/ullyses/vfts87/dr4/hlsp_ullyses_hst_stis_vfts87_e140m_dr4_cspec.fits to ./mastDownload/HLSP/hlsp_ullyses_hst_stis_vfts87_e140m/hlsp_ullyses_hst_stis_vfts87_e140m_dr4_cspec.fits ...
|==========================================| 650k/650k (100.00%)         0s
Downloading URL s3://stpubdata/hst/public/ody2/ody203010/ody203010_flt.fits to ./mastDownload/HST/ody203010/ody203010_flt.fits ...
|==========================================|  10M/ 10M (100.00%)         0s
Downloading URL s3://stpubdata/hst/public/ody2/ody203010/ody203010_raw.fits to ./mastDownload/HST/ody203010/ody203010_raw.fits ...
|==========================================| 8.4M/8.4M (100.00%)         0s
Downloading URL s3://stpubdata/hst/public/ody2/ody203010/ody203010_x1d.fits to ./mastDownload/HST/ody203010/ody203010_x1d.fits ...
|==========================================| 1.7M/1.7M (100.00%)         0s
Downloading URL s3://stpubdata/hst/public/ody2/ody219010/ody219010_flt.fits to ./mastDownload/HST/ody219010/ody219010_flt.fits ...

Note in the example above that 7 of the requested products are downloaded from the MAST Portal while the remaining half are downloaded from the s3 bucket as instructed. This is because HLSP data are not currently available in the cloud, so the default behavior is to fall back to the regular MAST download for such cases, which allows for a smoother download experience overall.

How do I know if my data is in the cloud?

Users can check if their data is in the cloud using either the get_cloud_uri or get_cloud_uris methods from astroquery.mast.Observations like so:

>>> Observations.enable_cloud_dataset(provider='AWS')
INFO: Using the S3 STScI public dataset [astroquery.mast.core]

>>> Observations.get_cloud_uris(filtered)
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_av39a_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_av454_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_av61_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_av81_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_vfts180_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_vfts440_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
WARNING: NoResultsWarning: Unable to locate file hlsp_ullyses_hst_stis_vfts87_e140m_dr4_cspec.fits. [astroquery.mast.cloud]
Out[95]:
[None,
 None,
 None,
 None,
 None,
 None,
 None,
 's3://stpubdata/hst/public/ody2/ody203010/ody203010_flt.fits',
 's3://stpubdata/hst/public/ody2/ody203010/ody203010_raw.fits',
 's3://stpubdata/hst/public/ody2/ody203010/ody203010_x1d.fits',
 's3://stpubdata/hst/public/ody2/ody219010/ody219010_flt.fits']

Brought to you by Jenny V. Medina