Another day another OpenStack integration – today I integrated 3PAR storage over iSCSI with HOS 1.1.1 for use with cinder block storage (a.k.a. volumes or persistent storage). As always please verify the latest hardware pre-requisites here – http://docs.hpcloud.com/#commercial/GA1/1.1commercial.install-GA-supportmatrix.html before you start.
Prerequisites
An assumption with this blog post is that you know nothing about 3PAR storage and there’s a storage administrator nearby to configure the 3PAR itself. They 3PAR admin will need to ensure that the Web Service API (WSAPI) is enabled and supply you with the configuration details [username, password, api endpoint, port]. They can enable this by SSH’ing onto the storage controller and configuring through the CLi – no rocket scientists required. You will also need at least one Common Provisioning Group (CPG) configured as the storage pool for cinder to carve Volumes from.
Both the management and iSCSI ip addresses will need to be accessible from the management LAN and the relevant ports opened on any firewalls between these networks. The netcat or nc tool detailed in my previous post will help if you have to verify connectivity between these networks.
The final pre-requisite is that you have a fully installed and configured HOS 1.1.1 system to integrate with 🙂
Integration
1. Register StoreServ to the cloud inventory
[Note: I used the –debug option for more details, this is not necessary]
# sirius --debug register-storeserv --name 7200_01 --hostname 10.223.148.158 --username storadm --password bananas --port 8080 --san-ip 10.223.148.158 --san-username storadm --san-password bananas --device-type iscsi --iscsi-ip 10.123.25.250 |& tee 3par_error.log
DEBUG (v2:77) Making authentication request to http://127.0.0.1:5000/v2.0/tokens
INFO (connectionpool:188) Starting new HTTP connection (1): 127.0.0.1
DEBUG (connectionpool:364) "POST /v2.0/tokens HTTP/1.1" 200 3860
DEBUG (iso8601:184) Parsed 2015-10-22T10:55:30Z into {'tz_sign': None, 'second_fraction': None, 'hour': u'10', 'daydash': u'22', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'30', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'10', 'day': None, 'minute': u'55'} with default timezone <iso8601.iso8601.Utc object at 0x7fcb3fc0c090>
DEBUG (iso8601:140) Got u'2015' for 'year' with default None
DEBUG (iso8601:140) Got u'10' for 'monthdash' with default 1
DEBUG (iso8601:140) Got 10 for 'month' with default 10
DEBUG (iso8601:140) Got u'22' for 'daydash' with default 1
DEBUG (iso8601:140) Got 22 for 'day' with default 22
DEBUG (iso8601:140) Got u'10' for 'hour' with default None
DEBUG (iso8601:140) Got u'55' for 'minute' with default None
DEBUG (iso8601:140) Got u'30' for 'second' with default None
DEBUG (http:110) curl -i -X POST -H 'X-Auth-Token: 6dd66fa3130f4ec38fdda6e43e3553be' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-siriusclient' -d '{"username": "storadm", "iscsi_ip": "10.123.25.250", "san_password": "bananas", "hostname": "10.223.148.158", "device_type": "iscsi", "san_username": "storadm", "password": "bananas", "san_ip": "10.223.148.158",
port": "8080", "name": "7200_01"}' http://10.123.25.14:9000/v1/block/device/storeserv
DEBUG (http:120)
HTTP/1.0 200 OK
date: Thu, 22 Oct 2015 06:55:31 GMT
content-length: 439
content-type: application/json; charset=UTF-8
server: WSGIServer/0.1 Python/2.7.8
{"storeserv": {"username": "storadm", "status": "registered", "san_password": "bananas", "deleted": false, "updated_at": "2015-10-22 06:55:31.262364", "device_type": "iscsi", "san_username": "storadm", "password": "bananas", "san_ip": "10.223.148.158", "port": "8080", "name": "7200_01", "iscsi_ip": "10.123.25.250", "created_at": "2015-10-22 06:55:31.262355", "hostname": "10.223.148.158", "id": "e29f3c58-7889-11e5-96e9-0017a477c444"}}
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| created_at | 2015-10-22 06:55:31.262355 |
| deleted | False |
| device_type | iscsi |
| hostname | 10.223.148.158 |
| id | e29f3c58-7889-11e5-96e9-0017a477c444 |
| iscsi_ip | 10.123.25.250 |
| name | 7200_01 |
| password | <SANITIZED> |
| port | 8080 |
| san_ip | 10.223.148.158 |
| san_password | bananas |
| san_username | storadm |
| status | registered |
| updated_at | 2015-10-22 06:55:31.262364 |
| username | storadm |
+--------------+--------------------------------------+
#
2. Register StoreServ CPG to the cloud inventory
First, list the available CPGs – be very careful that you select the correct one – as confucius says “measure twice, cut once” actually that may have been Tim the toolman.
# sirius storeserv-show e29f3c58-7889-11e5-96e9-0017a477c444 --cpg-list true +--------------------------------------------------------+------------------+ | UUID | Name | +--------------------------------------------------------+------------------+ | 137fcaf9-670b-43c2-89c5-1d28247f5847 | CPG_DEMO_Openstack | | 49671c1e-aee0-4527-9844-a384a4fd28c0 | CPG_ISCSI_R1 | | fab7065b-ca61-4346-9546-d8631741f5d8 | CPG_ISCSI_R5 | +--------------------------------------+------------------+ # Now register the correct CPG(s) # sirius register-cpg --cpgs 137fcaf9-670b-43c2-89c5-1d28247f5847 --storeserv-id e29f3c58-7889-11e5-96e9-0017a477c444 Successfully registered the StoreServ CPG(s)
Verify the results
# sirius cpg-list --storeserv-id e29f3c58-7889-11e5-96e9-0017a477c444 +--------------------------------------+------------------+------------+ | ID | Name | Status | +--------------------------------------+------------------+------------+ | 137fcaf9-670b-43c2-89c5-1d28247f5847 | CPG_DEMO_Openstack | registered | +--------------------------------------+------------------+------------+ #
3. Add StoreServ CPG as a backend for the cloud
# sirius add-storeserv-backend --storeserv-id e29f3c58-7889-11e5-96e9-0017a477c444 --cpg-id 137fcaf9-670b-43c2-89c5-1d28247f5847 --backend-name 3par_iscsi_1 +---------------------------+--------------------------------------------------------------+ | Property | Value | +---------------------------+--------------------------------------------------------------+ | backend_id | CPG_137fcaf9-670b-43c2-89c5-1d28247f5847 | | hp3par_api_url | https://10.223.148.158:8080/api/v1 | | hp3par_cpg | CPG_DEMO_Openstack | | hp3par_iscsi_chap_enabled | true | | hp3par_password | bananas | | hp3par_username | storadm | | iscsi_ip_address | 10.123.25.250 | | san_ip | 10.223.148.158 | | san_login | storadm | | san_password | bananas | | volume_backend_name | 3par_iscsi_1 | | volume_driver | cinder.volume.drivers.san.hp.hp_3par_iscsi.HP3PARISCSIDriver | +---------------------------+-------------------------------------------------------------+ #
Verify
# sirius cpg-show --storeserv-id e29f3c58-7889-11e5-96e9-0017a477c444 --cpg-id 137fcaf9-670b-43c2-89c5-1d28247f5847 --backend-name 3par_iscsi_1 The cinder.conf section preview for the selected CPG: [CPG_137fcaf9-670b-43c2-89c5-1d28247f5847] san_password=bananas hp3par_username=storadm volume_backend_name=3par_iscsi_1 san_login=storadm hp3par_api_url=https://10.223.148.158:8080/api/v1 volume_driver=cinder.volume.drivers.san.hp.hp_3par_iscsi.HP3PARISCSIDriver hp3par_password=bananas hp3par_cpg=CPG_DEMO_Openstack hp3par_iscsi_chap_enabled=true san_ip=10.223.148.158 iscsi_ip_address=10.123.25.250 #
List all registered backends
# sirius storeserv-backend-list +------------------------------------------+------------------+---------------------+ | Backend ID | CPG Name | Volume Backend Name | +------------------------------------------+------------------+---------------------+ | CPG_137fcaf9-670b-43c2-89c5-1d28247f5847 | CPG_DEMO_Openstack | 3par_iscsi_1 | +------------------------------------------+------------------+---------------------+ #
4. Preview the cinder configuration for the StoreServ CPG
This outputs the details that will be used to automagically (heat) update the cinder.conf files on the relevant nodes (controller & compute) once the overcloud update is run (step 6 below).
If there were more hours in the day I’d raise a ticket to have this function create a complete ‘cut-n-paste’ output that can be dropped directly into the configuration file rather than cut, paste, and potentially make a mistake whilst adding “3par :”.
# sirius storeserv-backend-list --format json
{
"DEFAULT": {
"enabled_backends": [
"CPG_137fcaf9-670b-43c2-89c5-1d28247f5847"
]
},
"CPG_137fcaf9-670b-43c2-89c5-1d28247f5847": {
"san_password": "bananas",
"hp3par_username": "storadm",
"volume_backend_name": "3par_iscsi_1",
"san_login": "storadm",
"hp3par_api_url": "https://10.223.148.158:8080/api/v1",
"volume_driver": "cinder.volume.drivers.san.hp.hp_3par_iscsi.HP3PARISCSIDriver",
"hp3par_password": "bananas",
"hp3par_cpg": "CPG_DEMO_Openstack",
"hp3par_iscsi_chap_enabled": "true",
"san_ip": "10.223.148.158",
"iscsi_ip_address": "10.123.25.250"
}
}
#
5. Update Overcloud configuration JSON (a.k.a. kvm-custom-ips.json)
IMPORTANT: The official documentation is confusing here as it refers to a /root/overcloud-config.json file but this file doesn’t exist if you installed HOS1.1.1 following the official documentation 🙂 The file it should refer to is the /root/tripleo/configs/kvm-custom-ips.json that’s configured when building the cloud.
This is what the file looks like before any amendments
{
"cloud_type": "KVM",
"vsa_scale": 0,
"vsa_ao_scale": 0,
"so_swift_storage_scale": 0,
"so_swift_proxy_scale": 0,
"compute_scale": 2,
"bridge_interface": "em1",
"virtual_interface": "eth0",
"fixed_range_cidr": "172.0.100.0/24",
"control_virtual_router_id": "157",
"baremetal": {
"network_seed_ip": "10.123.25.13",
"network_cidr": "10.123.25.0/24",
"network_gateway": "10.123.25.1",
"network_seed_range_start": "10.123.25.14",
"network_seed_range_end": "10.123.25.34",
"network_undercloud_range_start": "10.123.25.35",
"network_undercloud_range_end": "10.123.25.135"
},
"neutron": {
"public_interface_raw_device": "eth0",
"overcloud_public_interface": "vlan1174",
"undercloud_public_interface": "eth0"
},
"dns": {
"overcloud_server": "10.123.121.10",
"undercloud_server": "10.123.121.10",
"seed_server": "10.123.121.10"
},
"ntp": {
"overcloud_server": "10.123.226.30",
"undercloud_server": "10.123.226.30",
"seed_server": "10.123.226.30"
},
"floating_ip": {
"start": "10.123.27.10",
"end": "10.123.27.250",
"cidr": "10.123.27.0/24"
},
"svc": {
"interface": "vlan1173",
"interface_default_route": "10.123.26.1",
"allocate_start": "10.123.26.10",
"allocate_end": "10.123.26.250",
"allocate_cidr": "10.123.26.0/24",
"overcloud_bridge_mappings": "svcnet1:br-svc",
"overcloud_flat_networks": "svcnet1",
"customer_router_ip": "10.123.25.1"
},
"codn": {
"undercloud_http_proxy": "http://fw1.osh.gosh:8080",
"undercloud_https_proxy": "http://fw1.osh.gosh:8080",
"overcloud_http_proxy": "http://fw1.osh.gosh:8080",
"overcloud_https_proxy": "http://fw1.osh.gosh:8080"
}
}
Following the amendments the file should look like this
# cat tripleo/configs/kvm-custom-ips.json
{
"cloud_type": "KVM",
"vsa_scale": 0,
"vsa_ao_scale": 0,
"so_swift_storage_scale": 0,
"so_swift_proxy_scale": 0,
"compute_scale": 2,
"bridge_interface": "em1",
"virtual_interface": "eth0",
"fixed_range_cidr": "172.0.100.0/24",
"control_virtual_router_id": "157",
"baremetal": {
"network_seed_ip": "10.123.25.13",
"network_cidr": "10.123.25.0/24",
"network_gateway": "10.123.25.1",
"network_seed_range_start": "10.123.25.14",
"network_seed_range_end": "10.123.25.34",
"network_undercloud_range_start": "10.123.25.35",
"network_undercloud_range_end": "10.123.25.135"
},
"neutron": {
"public_interface_raw_device": "eth0",
"overcloud_public_interface": "vlan1174",
"undercloud_public_interface": "eth0"
},
"dns": {
"overcloud_server": "10.123.121.10",
"undercloud_server": "10.123.121.10",
"seed_server": "10.123.121.10"
},
"ntp": {
"overcloud_server": "10.123.226.30",
"undercloud_server": "10.123.226.30",
"seed_server": "10.123.226.30"
},
"floating_ip": {
"start": "10.123.27.10",
"end": "10.123.27.250",
"cidr": "10.123.27.0/24"
},
"svc": {
"interface": "vlan1173",
"interface_default_route": "10.123.26.1",
"allocate_start": "10.123.26.10",
"allocate_end": "10.123.26.250",
"allocate_cidr": "10.123.26.0/24",
"overcloud_bridge_mappings": "svcnet1:br-svc",
"overcloud_flat_networks": "svcnet1",
"customer_router_ip": "10.123.25.1"
},
"3par": {
"DEFAULT": {
"enabled_backends": [
"CPG_137fcaf9-670b-43c2-89c5-1d28247f5847"
]
},
"CPG_137fcaf9-670b-43c2-89c5-1d28247f5847": {
"san_password": "bananas",
"hp3par_username": "storadm",
"volume_backend_name": "3par_iscsi_1",
"san_login": "storadm",
"hp3par_api_url": "https://10.223.148.158:8080/api/v1",
"volume_driver": "cinder.volume.drivers.san.hp.hp_3par_iscsi.HP3PARISCSIDriver",
"hp3par_password": "bananas",
"hp3par_cpg": "CPG_DEMO_Openstack",
"hp3par_iscsi_chap_enabled": "true",
"san_ip": "10.223.148.158",
"iscsi_ip_address": "10.123.25.250"
}
},
"codn": {
"undercloud_http_proxy": "http://fw1.osh.gosh:8080",
"undercloud_https_proxy": "http://fw1.osh.gosh:8080",
"overcloud_http_proxy": "http://fw1.osh.gosh:8080",
"overcloud_https_proxy": "http://fw1.osh.gosh:8080"
}
}
6. Source the above file and update the Overcloud configuration
From the seedvm – the first of TripleO’s clouds source the new configuration file
source tripleo/tripleo-incubator/scripts/hp_ced_load_config.sh tripleo/configs/kvm-custom-ips.json
And now reconfigure the Overcloud heat-templates and apply them using the HOS installer
bash -x tripleo/tripleo-incubator/scripts/hp_ced_installer.sh --update-overcloud |& tee 3ParIntegration.log
The output looks like this:
+ set -eu
+ set -o pipefail
++ basename tripleo/tripleo-incubator/scripts/hp_ced_installer.sh
+ SCRIPT_NAME=hp_ced_installer.sh
++ dirname tripleo/tripleo-incubator/scripts/hp_ced_installer.sh
+ SCRIPT_HOME=tripleo/tripleo-incubator/scripts
+ umask 077
+ export LANG=C
+ LANG=C
++ awk -F= '{print $1}'
++ grep '^LC_'
++ env
+ export SKIP_INSTALL_SEED=0
+ SKIP_INSTALL_SEED=0
+ export SKIP_INSTALL_UNDERCLOUD=0
+ SKIP_INSTALL_UNDERCLOUD=0
+ export SKIP_INSTALL_OVERCLOUD=0
+ SKIP_INSTALL_OVERCLOUD=0
+ export UPDATE_UNDERCLOUD=0
+ UPDATE_UNDERCLOUD=0
+ export UPDATE_OVERCLOUD=0
+ UPDATE_OVERCLOUD=0
+ export SKIP_DEMO=0
+ SKIP_DEMO=0
+ export POWER_OFF_NODES_WAIT_INTVL=15
+ POWER_OFF_NODES_WAIT_INTVL=15
+ export TRIPLEO_ROOT=/root/tripleo
+ TRIPLEO_ROOT=/root/tripleo
+ CE_FLAG=/root/tripleo/HP_Helion_OpenStack
+ BUILD_FLAG=/root/tripleo/HP_Helion_Version
+ '[' -e /root/tripleo/HP_Helion_Version ']'
++ cat /root/tripleo/HP_Helion_Version
+ VERSION='jenkins-release-ee-on-hlinux-hp-ee-stable-1.0-100: Thu May 14 07:40:45 UTC 2015'
+ export 'BUILD_TAG=jenkins-release-ee-on-hlinux-hp-ee-stable-1.0-100: Thu May 14 07:40:45 UTC 2015'
+ BUILD_TAG='jenkins-release-ee-on-hlinux-hp-ee-stable-1.0-100: Thu May 14 07:40:45 UTC 2015'
+ echo
.......SKIPPING LOTS OF OUTPUT.......
"value": "true"}, {"option": "rpc_response_timeout", "value": "120"}, {"option": "control_exchange", "value": "cinder"}, {"option": "notification_driver", "value": "messaging"}, {"comment": "Common hostname to avoid singleton limitation of Cinder volume manager", "option": "host", "value": "ha-volume-manager"}, {"option": "enabled_backends", "value": "CPG_137fcaf9-670b-43c2-89c5-1d28247f5847"}]}, {"section": "lvm-1", "values": [{"option": "volume-group", "value": "cinder-volumes"}, {"option": "lvm_type", "value": "thin"}, {"option": "volume_driver", "value": "cinder.volume.drivers.lvm.LVMISCSIDriver"}, {"option": "volume_backend_name", "value": "LVM_iSCSI"}]}, {"section": "CPG_137fcaf9-670b-43c2-89c5-1d28247f5847", "values": [{"option": "san_password", "value": "bananas"}, {"option": "hp3par_username", "value": "storadm"}, {"option": "volume_backend_name", "value": "3par_iscsi_1"}, {"option": "san_login", "value": "storadm"}, {"option": "hp3par_api_url", "value": "https://10.223.148.158:8080/api/v1"}, {"option": "volume_driver", "value": "cinder.volume.drivers.san.hp.hp_3par_iscsi.HP3PARISCSIDriver"}, {"option": "hp3par_password", "value": "bananas"}, {"option": "hp3par_cpg", "value": "CPG_DEMO_Openstack"}, {"option": "hp3par_iscsi_chap_enabled", "value": "true"}, {"option": "san_ip", "value": "10.223.148.158"}, {"option": "iscsi_ip_address", "value": "10.123.25.250"}]}]}, "swift": {"config": [{"section": "storage-policy:0", "values": [{"option": "name", "value": "Policy-0"}, {"option": "default", "value": "yes"}]}]}, "ceilometer_pipeline": {"sources": [{"interval": 604800, "meters": [{"name": "instance"}, {"name": "image"}, {"name": "image.size"}, {"name": "image.upload"}, {"name": "image.delete"}, {"name": "volume"}, {"name": "volume.size"}, {"name": "snapshot"}, {"name": "snapshot.size"}, {"name": "ip.floating"}, {"name": "network.*"}, {"name": "compute.instance.create.end"}, {"name": "compute.instance.delete.end"},
.......SKIPPING LOTS OF OUTPUT.......
UNDERCLOUD Endpoint: "http://10.123.25.14:5000/v2.0"
UNDERCLOUD IP: "10.123.25.14"
OVERCLOUD Endpoint: "https://10.123.25.35:5000/v2.0/"
OVERCLOUD IP: "10.123.25.35"
++ date
+ echo 'HP - completed - Thu Oct 22 08:49:18 UTC 2015'
HP - completed - Thu Oct 22 08:49:18 UTC 2015
+ exit 0
“exit 0” is Success in our case.
7. Create VolumeType with Extra Spec Key Value Pair ‘volume_backend_name’ and ‘3par_iscsi_1’
Now you can leave the CLi and jump onto Horizon (Overcloud) for the final step of configuring a VolumeType and associating it with a storage backend. See http://docs.hpcloud.com/#commercial/GA1/1.1commercial.map.volumetype.html
You should now have the ability to create Volumes with cinder on your 3Par – Costly Persistent Storage – Look into Ceph the open source alternative for more affordable cloud scale block/volume solutions.
Cumulus Chatter 
Leave a comment