Lesson 1: What is the Archive Storage solution?

The low-tier archive storage is a high capacity storage designed to store data which is infrequently used but needs to be stored safely and sustainably, for occasional access. By providing this on tape media we are able to provide this at much lower cost than disk-based storage. The trade-off is that recalling the data from tape is not instantaneous, but depending on the quantity of data recalled, there will be a waiting period to make the files available, hence the low tier description, describing the latency to access the files. This recall action can be initiated by the end-user/data owner, via the Globus file transfer service.

The idea is somewhat analogous to cloud archive services such as Amazon Glacier and Microsoft Azure Cold storage. However this service has better pricing and quicker file access than the standard cloud archive offerings, which may take a few hours to retrieve. Depending on the queue and file size, files on the QMUL archive are typically available within a few minutes to an hour.

Each data store has different qualities depending on its purpose. You can read more about when to use each one in more detail on the HPC documentation site.

Some Research groups have large amounts of data that they do not need fully accessible all of the time on more expensive, high performance storage - for example, data that has already been analysed, but needs to be kept for a few more years just in case it is needed again. Basically, datasets currently stored on the research data store that might only be used once or twice a year. For such cases, moving data onto the archive will save annual storage costs.

Note that, by design, data residing on the archive storage cannot be processed directly on the HPC nodes. It must be recalled from archive and copied somewhere.

The method for copying the data to the tape archive and subsequent data recall is done via the Globus tool. We recommend you follow these preliminary lessons to familiarise yourself with the basic Globus file operations before proceeding with these specific tutorials.

The archive is a paid service that can be requested via email to its-research-support@qmul.ac.uk , and we will collect a few details from you, including names of users who will be authorised to access the archive.