Dane na klastrze Eagle
New data structure.
Home directory: is a limited data space with a small capacity limit of a few hundred megabytes. In practice, it is a sack for project directories.
'Project directory Each user with an active computing grant (scientific or commercial) on the cluster in his 'Home directory' has an automatically created directory named '<grant_number>' (e.g. pl0001-01) for each grant in which the user participates. This directory contains two important subdirectories:
- archive' - is a space for storing data that is not actively used, such as processed calculation results, etc. This space is slower than project_data and scratch, but has a much larger size - it is possible to store tens, or (if necessary), hundreds of TB of data;
- project_data - is a directory shared among all users of a given grant. Data located in all subdirectories of this directory are protected from accidental deletion by an automatic backup mechanism. PSNC provides a guaranteed data space for this directory in the size granted under the grant application or commercial contract. This limit can only be used by users. Within this directory, users can freely create and delete files/directories and manipulate access rights. The default space available in this directory is 5TB;
- scratch - is 'dedicated' space for a given grant to run calculations/storage of input data. The usage rules are identical to the currently used scratch space. The default size of this space is 10TB.
The data available in the 'project_data' directory will be available for the entire duration of the grant extended by some additional time to archive the data or move it to the space of the new grant. Currently, data is made available for a period of 6 months after the end of the grant, plus an additional 6 months during which data can be retrieved upon request. The new data structure is as follows:
-> ~<user_name> : home directory --> grant_X ---> project_data : shared data space for grant_X ---> scratch : shared data space for data at computation time for grant_x ---> archive : shared space for archive data for grant_X --> grant_y ---> project_data : shared data space for grant_Y ---> scratch : shared space for data at the time of calculation for grant_Y ---> archive : shared space for archive data for grant_Y.
In the future, there will be additional directories providing, among other things, easier access to the archiving system or the possibility of exchanging data with the box.pionier.net.pl service.
Note: the project_data and scratch directories are symbolic links to the actual storage system mount location. Users are asked to use only relative paths (e.g. /home/users/<user_name>/grant_id/project_data), not physical mount paths as these may change. The new data structure allows us to dynamically move grants between different storage systems "on the fly," so to speak, so there is no guarantee that the data will be stored on the same physical system throughout the life of the grant.
The storage systems currently in use are being phased out starting at the end of 2021. - the /tmp/lustre directory will disappear and space in the home directory will be significantly reduced - the 'project_data' and 'scratch' directories of grants should become the main storage location. Users will be informed of any planned work at least two months in advance to allow data migration.
To check how much space a given directory (archive, project_data, scratch) occupies and what limint it has, use the following commands:
How to check quotas and resource limtis for project_data and scratch: When you log-in a banner with information about usage of limits (both storage and cpu/gpu time) are displayed. Please bear in mind that these values refresh every 24h. You can also check these values in the service details on the portal https://pcss.plcloud.pl/