ASYNCHRONOUS DATA REQUESTS (UP TO 50 GB)

    An asynchronous request takes the search criteria and instead of starting the download straight away, it creates the tar.gz package in the background and then sends an email when the process is complete. This method is less susceptible to timeouts or slow internet than a direct request. The package is available for download for 2 days and then deleted from our server to save space.

     

    If the requested data package is greater in volume than 10 GB (compressed), an asynchronous data request is required. There are two changes necessary compared to the Direct Data Request:

    1. Authentication is required - either in the web interface, or with wget, curl or a programming language.

    2. The keyword 'RETRIEVAL_ACCESS' is added, with the value 'DEFERRED' (note that DEFERRED is case sensitive)

    After this request is made, the SOAR server will assemble the package, then send an email to the registered address when it is ready for collection - so make sure that the email registered to the SOAR user ID is up to date.

    Alternatively, rather than waiting for an email, details are provided below on how to check the status of the asynchronous job, how to tell if it is complete and ready for download, and where to go to get the completed package.

     

    If you want to download large portions of the archive, e.g., all files for one SOOP, this is how it should be done. 

     

    Log In

    You need to have registered with us and then authenticate at: 

    https://soar.esac.esa.int/soar-sl-tap/login

     

    THE BASE

    The details of the request need to be built on the following base:

    https://soar.esac.esa.int/soar-sl-tap/data?RETRIEVAL_TYPE=PRODUCT&RETRIEVAL_ACCESS=DEFERRED&

    This is the same as for the direct download requests, but with 'RETRIEVAL_ACCESS=DEFERRED' added. 

     

    After this, add the data_item_id: 

    https://soar.esac.esa.int/soar-sl-tap/data?retrieval_type=PRODUCT&&RETRIEVAL_ACCESS=DEFERRED&data_item_id=solo_L2_epd-ept-north-rates_20200820,solo_L2_epd-ept-south-rates_20200820&product_type=SCIENCE

    or ADQL query:

    https://soar.esac.esa.int/soar-sl-tap/data?retrieval_type=PRODUCT&&RETRIEVAL_ACCESS=DEFERRED&QUERY=SELECT+filepath,filename+FROM+soar.v_sc_repository_file+WHERE+filename='solo_L2_epd-sis-a-rates-slow_20200615_V05.cdf'

    just like for a direct data request.

     

    Then wait (some requests can take a while) and when it's ready for collection, an email will be sent with the address to pick up the package:

     

    Example of the email sent when package is ready for collection:

    The XML Response

    Performing the request will generate a response in the form of an XML file containing details of the request. Putting the request into the address bar of a browser will generate the XML in the browser as shown below. However, if you wish to perform asynchronous requests within a script (particularly if you don't want to rely on the email for the collection address), the contents of the XML will need to be used for the status and collection address for the package.

    The following screenshot shows the XML response to an asynchronous request made in the address bar of a browser. If you are not familiar with reading XML files, the image below shows that the content is arranged in fields, with a start and an end (marked by '/'); you may need developer tools to be active in your browser to view it like this.

    For example, the first field highlighted in this screenshot is for the <uws:phase>:

    <uws:phase>EXECUTING</uws:phase>

    in this case the value is EXECUTING, which means that the package is still being assembled. 

    The status of the job can be checked by reloading the page; note that this is also the URL value inside the <uws:parameter id="email_base_url">, the second field highlighted below; for example:

    <uws:parameter id="email_base_url">
    <![CDATA[ http://soar.esac.esa.int/soar-sl-tap/tap/async/1651071708120OPE ]]>
    </uws:parameter>

    Reload the page until the <uws:phase> changes to COMPLETED (see image at bottom of page):

    <uws:phase>COMPLETED</uws:phase>

    then there will be a value inside the <uws:results> field at the bottom of the XML (see image at bottom of page), e.g.,:

    <uws:results>
    <uws:result id="hmiddlet1651071708155" xlink:type="simple" xlink:href="http://soar.esac.esa.int/soar-sl-tap/tap/async/1651071708120OPE/results/hmiddlet1651071708155" size="953398938"/>
    </uws:results>

    The data will download from the URL inside the uws:results field.

    The COMPLETED package XML: