Large Custom Images – Uploading Files > 5GB to Fujitsu K5 Object Storage (Swift)

In a previous blog post I detailed a simple process for uploading custom images to the Fujitsu K5 IaaS platform. One challenge that I overlooked is large custom image sizes which I’ll address here.

Background: OpenStack Swift objects have a maximum size limitation of 5GB. However the Swift containers can have many thousands of these objects. Files that are larger than 5GB need to be split into a subset of smaller files before being uploaded to the container. When all the component files have been uploaded a zero byte file with a ‘manifest’ header is uploaded to the container. This header is composed of the container name and prefix used to build the component file names. When this zero byte file is referenced through the Swift API all the component files are concatenated and the original large file is downloaded.

The following python script will upload custom images smaller than 1 GB directly into K5’s object storage. Larger files are split into 1GB chunks before uploading to the object storage. The default 1GB size can be changed by using the command line parameters.

Once the image is uploaded it is then registered with the the default K5 project. The image must now be shared with other member projects that wish to consume this image – this will be shared in the next post.

Prerequisites : The script relies on a settings file, k5contractsettings.py, which must contain all your contract details and be placed in the same directory  – for example:

Image Upload Example Script

Example script output:

C:\Users\landg\>python K5ImageUpload.py -i "c:\Users\landg\Downloads\cirrosuploadtest.vmdk" -c uploaddemo12 -n "h
ello k5 milti image"  -t ubuntu -p NotUsed -s 5242880

Creating new container : uploaddemo12
31c0f27e562c4b3089a546c175c144e4
https://objectstorage.uk-1.cloud.global.fujitsu.com/v1/AUTH_eadb882573ac40b1b101
eac93009a313/uploaddemo12

Created new container : uploaddemo12

---------- Starting multi-part file upload  to K5 object storage ------

Uploaded Package - 1
Uploaded Package - 2
Uploaded Package - 3
Uploaded Package - 4

Uploaded Package - 4

---------- Finished multi-part file upload to K5 object storage ------


---------- List container contents K5 object storage start ------

<Response [200]>

---------- List container contents K5 object storage end ------


---------- Registering image with K5 ------

{u'import_id': u'6a0b58c5-bcda-4a64-919e-23f06b8338ad'}

---------- K5 Image import_id : 6a0b58c5-bcda-4a64-919e-23f06b8338ad


---------- Check import status ----------


{u'conversion': True, u'name': u'hello k5 milti image', u'container_format': u'b
are', u'min_ram': u'0', u'ovf_location': u'', u'disk_format': u'raw', u'domain_n
ame': u'YssmW1yI', u'location': u'/v1/AUTH_eadb882573ac40b1b101eac93009a313/uplo
addemo12/cirrosuploadtest.vmdk', u'min_disk': u'0', u'progress': u'0', u'os_type
': u'ubuntu', u'password': u'*', u'user_name': u'landg', u'id': u'70a38639-f819-
4375-b3d2-cfc99c2a148e', u'import_status': u'queued'}

---------- Check import status ----------

{u'conversion': True, u'name': u'hello k5 milti image', u'container_format': u'b
are', u'min_ram': u'0', u'ovf_location': u'', u'disk_format': u'raw', u'domain_n
ame': u'YssmW1yI', u'location': u'/v1/AUTH_eadb882573ac40b1b101eac93009a313/uplo
addemo12/cirrosuploadtest.vmdk', u'min_disk': u'0', u'progress': 0, u'os_type':
u'ubuntu', u'password': u'*', u'user_name': u'landg', u'id': u'70a38639-f819-437
5-b3d2-cfc99c2a148e', u'import_status': u'processing'}

---------- Check import status ----------

{u'container_format': u'bare', u'min_ram': 0, u'updated_at': u'2016-10-08T17:24:
38Z', u'file': u'/v2/images/70a38639-f819-4375-b3d2-cfc99c2a148e/file', u'owner'
: u'eadb882573ac40b1b101eac93009a313', u'id': u'70a38639-f819-4375-b3d2-cfc99c2a
148e', u'size': 41126400, u'conversion': True, u'self': u'/v2/images/70a38639-f8
19-4375-b3d2-cfc99c2a148e', u'disk_format': u'raw', u'domain_name': u'YssmW1yI',
 u'location': u'/v1/AUTH_eadb882573ac40b1b101eac93009a313/uploaddemo12/cirrosupl
oadtest.vmdk', u'progress': 100, u'user_name': u'landg', u'schema': u'/v2/schema
s/image', u'status': u'active', u'import_status': u'succeeded', u'tags': [], u'v
isibility': u'private', u'BaseImageId': u'415b3a0a513aebc27d34c68bd8cdae8c', u'm
in_disk': 0, u'password': u'*', u'name': u'hello k5 milti image', u'created_at':
 u'2016-10-08T17:24:32Z', u'ovf_location': u'', u'fcx.centos': u'true', u'protec
ted': False, u'os_type': u'ubuntu'}
End of Import Process - Import status >>>  succeeded

Happy Stacking!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s