Pycosio (Python Cloud Object Storage I/O)¶
Pycosio brings standard Python I/O to cloud objects by providing:
- Abstract classes of Cloud objects with the complete
- Features equivalent to the standard library (
shutil) for seamlessly managing cloud objects and local files.
These functions are source agnostic and always provide the same interface for all files from cloud storage or local file systems.
Buffered cloud objects also support the following features:
- Buffered asynchronous writing of any object size.
- Buffered asynchronous preloading in reading mode.
- Write or read lock depending on memory usage limitation.
- Maximization of bandwidth using parallels connections.
Example of code:
import pycosio # Open an object on AWS S3 as text for reading with pycosio.open('s3://my_bucket/my_object.txt', 'rt') as file: text = file.read() # Open an object on AWS S3 as binary for writing with pycosio.open('s3://my_bucket/my_object.bin', 'wb') as file: file.write(b'binary_data') # Copy file from the local file system to OpenStack Swift pycosio.copy( 'my_file', 'https://objects.my_cloud.com/v1/12345678912345/my_container/my_object') # Get size of a file over internet pycosio.getsize('https://www.example.org/any_object') >>> 956
Supported Cloud storage¶
Pycosio is compatible with the following cloud objects storage services:
- Alibaba Cloud OSS
- Amazon Web Services S3
- Microsoft Azure Blobs Storage
- Microsoft Azure Files Storage
- OpenStack Swift
Pycosio can also access any publicly accessible file via HTTP/HTTPS (Read only).
Cloud object storage is not file systems and has the following limitations:
- Cloud objects are not seekable in write mode.
- Cloud objects must be written entirely at once.
- Cloud objects are not locked when accessed.
- The cloud object attributes available are more limited.