DataManager schedules and manages the secure transfer and storage of data sets needed for large scale computations. It seamlessly automates data archival and retrieval from lower-cost cloud storage solutions, such as Amazon Glacier. This enables users to more quickly and cost-effectively conduct a wide range of compute and data-intensive workloads in areas such as life sciences, financial services, manufacturing, academia and energy. It can be used to automatically move data to utility HPC environments using a variety of open source and third-party transfer protocols and appliances. Features include data elasticity, efficiently moving data from endpoint to endpoint, such as a local file system to Glacier or a remote file system, based on the specific date and time it is needed; data awareness and usage chargeback; and client-side security and encryption, with local control of encryption and key management for data in transit and at rest in Amazon Glacier.