Skip to main content

User Guide Introduction

Welcome to the StageIT User Guide. This guide describes the functionality that this application offers and how can it can be used. In the menu on the left, you will find various topics that you can explore. If you have any questions, remarks, or bug reports, please use the SDC Helpdesk.

What is StageIT?​

LOFAR data is stored on a tape infrastructure at the LOFAR Long Term Archive (LTA) locations. Currently, the LTA locations are: SARA, Juelich and Poznan. Such facilities, use DCACHE (developed by CERN) to mediate the access from the external users to the data itself.

In particular, the data has two possible states: either is stored on tape or it is staged temporarily to be downloaded or accessed on a disk. Accessing the data is not trivial, and this process has been abstracted away into the StageIT application. Using this application, you can easily create requests to stage data and receive notifications when your stage request has been completed.

Differences between StageIT and the old stager service​

In principle StageIT should support all functionality the old stager supported. Below is a list of important differences between the old stager service and StageIT.

  • For StageIT, users will need to log in with their institutional credentials through the new Federated AAI system that will be used by all LOFAR user-facing tooling (SURF SRAM). We will be changing the logins for the LTA to this as well but for the time being this may mean a user staging data from the stager needs to log in twice.
  • For the Stager API, the implication is that users will need to request a token for automated access. Please refer to the documentation for more information
  • The new stager has a web interface that can be used to monitor staging requests.
  • Some functionality in the lofar_stager_api client library has been deprecated due to implementation details of the new stager. We refer to the lofar_stager_api README for details.
  • StageIT uses the webdav protocol for staging operations, using macaroons for authentication and authorization, due to the pending (December) deprecation of the SRM protocol at SURF. Refer to the documentation for details.
  • Stage requests can still be created with SRM urls (SURLS). StageIT will convert these to webdav URLS before sending the request to the LTA site.
  • Some SURLS in the LTA catalogue are missing a port number (e.g. 8443). StageIT will 'fix' these by adding the required port number, before saving the url to the StageIT database.
  • StageIT handles missing files gracefully. I.e. if any URLS in a stage request point to files which do not exist, the stage request continues staging the rest of the files, and finishes with 'partial success' status. The list of missing files is visible in the StageIT UI or API.