Installing and Configuring the Cloud Model
The first thing that we need to do is copy over one of the example cloud templates to use as a starting point. If your deployment model aligns exactly with the example then you will have only a few files to modify before building your cloud.
However, in the interest of showing how one could modify the model significantly I’ve decided to add another network and we will use it to provide separation for the Ceph OSD replication traffic.
Copy the ceph example configuration files into the my_cloud directory for editing.
cp -r ~/helion/examples/entry-scale-kvm-ceph/* ~/helion/my_cloud/definition/
These configuration files represent the following model:
However, we need to amend the model to accommodate the new replication network as shown below:
It’s now time to jump to your editor of choice, Komodo in my case, to start remodelling the example template files to match my requirements.
Add a remote connection in the editor to the Helion Lifecycle Manager (controller0).
You can edit the files in whatever sequences is convenient to you.
Please read the extensive online documentation that gives a detailed explanation of all the yaml files used to configure the cloud platform here.
cloudConfig.ym
I like to start with the simplest one, cloudConfig.yml
In this file we find the settings for the cloud name, hostname prefixes, ntp server details, dns server details, smtp settings and some general firewall settings.
The following values were changed from the defaults:
cloud name : allthingscloud
ntp : 172.16.1.5
dns : 172.16.1.5
Note: Ensure to quote both the NTP and DNS IP addresses. The example file does not have quotes on the DNS nameservers.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 cloud: name: allthingscloud # The following values are used when # building hostnames hostname-data: host-prefix: helion member-prefix: -m # List of ntp servers for your site ntp-servers: - "172.16.1.5" # - "ntp-server2" # dns resolving configuration for your site # refer to resolv.conf for details on each option dns-settings: nameservers: - "172.16.1.5" # - name-server2 # - name-server3 # # domain: sub1.example.net # # search: # - sub1.example.net # - sub2.example.net # # sortlist: # - 192.168.160.0/255.255.240.0 # - 192.168.0.0 # # # option flags are ':' to enable, remove to unset # # options with values are ':' to set # # options: # debug: # ndots: 2 # timeout: 30 # attempts: 5 # rotate: # no-check-names: # inet6: smtp-settings: # server: mailserver.examplecloud.com # port: 25 # timeout: 15 # These are only needed if your server requires authentication # user: # password: # Generate firewall rules firewall-settings: enable: true # log dropped packets logging: true
Control_plane.yml
This file defines all the failure zones, clusters, service components and resources. A quick inspection of this file reveals that the ‘lifecycle-manager’ service component is under the CONTROLLER-ROLE – in other words we have combined Controller/Deployer model.
The only parameter that I changed in this yaml file is the min-count for the compute resources from 0 to 2. I am going to deploy 2 compute nodes at build time.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 control-planes: - name: control-plane-1 control-plane-prefix: cp1 region-name: region1 failure-zones: - AZ1 - AZ2 - AZ3 common-service-components: - logging-producer - monasca-agent - freezer-agent - stunnel - lifecycle-manager-target clusters: - name: cluster1 cluster-prefix: c1 server-role: CONTROLLER-ROLE member-count: 3 allocation-policy: strict service-components: - lifecycle-manager - ntp-server - swift-ring-builder - mysql - ip-cluster - apache2 - keystone-api - keystone-client - rabbitmq - glance-api - glance-registry - glance-client - cinder-api - cinder-scheduler - cinder-volume - cinder-backup - cinder-client - nova-api - nova-scheduler - nova-conductor - nova-console-auth - nova-novncproxy - nova-client - neutron-server - neutron-ml2-plugin - neutron-vpn-agent - neutron-dhcp-agent - neutron-metadata-agent - neutron-openvswitch-agent - neutron-client - horizon - sherpa-api - swift-proxy - memcached - swift-account - swift-container - swift-object - swift-client - heat-api - heat-api-cfn - heat-api-cloudwatch - heat-engine - heat-client - openstack-client - ceilometer-api - ceilometer-collector - ceilometer-agent-central - ceilometer-agent-notification - ceilometer-expirer - ceilometer-common - ceilometer-client - zookeeper - kafka - vertica - storm - monasca-api - monasca-persister - monasca-notifier - monasca-threshold - monasca-client - logging-server - ops-console-web - ops-console-monitor - cmc-service - freezer-api - ceph-monitor resources: - name: compute resource-prefix: comp server-role: COMPUTE-ROLE allocation-policy: any min-count: 2 service-components: - ntp-client - nova-compute - nova-compute-kvm - neutron-l3-agent - neutron-metadata-agent - neutron-openvswitch-agent - neutron-lbaasv2-agent - name: osd resource-prefix: ceph server-role: OSD-ROLE allocation-policy: strict min-count: 3 service-components: - ntp-client - ceph-osd
servers.yml
This file needs all the details of the physical servers that are going to be used during the deployment.
- baremetal subnet in this deployment is also the management network : 172.16.60.10/24
- id should be unique to each server
- role relates to the server role and determines what components and services will be deployed on the server –> linked to server_roles.yml file
- ip-addr is the address that you would like the server to have on the MANAGEMENT network. Controller1 already had this ip address assigned as it is also the deployer node (Helion Lifecycle Manager) in this scenario
- server-group can be used to map to the physical layout of the servers to define failure hardware domains –> linked to server_groups.yml file
- nic-mappings defines the hardware address mapping of the physical nics -> linked to nic_mappings.yml file. Anyone who has worked with Linux networking over the past decade will appreciate the purpose of this file. Linux has a “feature” that can cause physical nics to be mapped to random different software interfaces (e.g. eth0, eth1, em59 etc) upon reboot. This file ‘hardwires’ the physical to software interface mappings for a consistent os reboot experience
- ilo-ip is the address of the IPMI interface on the physical server
- ilo-password – STOP now and go back to AZURE if you need an explanation for this parameter
- ilo-user – see above
- mac-address is the mac address of the PXE interface. In this deployment we’ll be PXE booting over the MANAGEMENT network. NOTE: THIS IS NOT THE MAC ADDRESS OF THE iLO INTERFACE
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 baremetal: # NOTE: These values need to be changed to match your environment. # Define the network range that contains the ip-addr values for # the individual servers listed below. subnet: 172.16.60.0 netmask: 255.255.255.0 servers: # NOTE: Addresses of servers need to be changed to match your environment. # # Add additional servers as required # # Controllers - id: controller1 ip-addr: 172.16.60.10 role: CONTROLLER-ROLE server-group: RACK1 nic-mapping: HP-SL230-4PORT ilo-ip: 172.30.0.45 ilo-password: "bananas" ilo-user: Administrator mac-addr: 8c:dc:d4:b5:cc:d0 - id: controller2 ip-addr: 172.16.60.11 role: CONTROLLER-ROLE server-group: RACK2 nic-mapping: HP-SL230-4PORT ilo-ip: 172.30.0.46 ilo-password: "bananas" ilo-user: Administrator mac-addr: 8c:dc:d4:b5:c6:40 - id: controller3 ip-addr: 172.16.60.12 role: CONTROLLER-ROLE server-group: RACK3 nic-mapping: HP-SL230-4PORT ilo-ip: 172.30.0.47 ilo-password: "bananas" ilo-user: Administrator mac-addr: 8c:dc:d4:b5:c9:74 # Compute Nodes - id: compute1 ip-addr: 172.16.60.13 role: COMPUTE-ROLE server-group: RACK1 nic-mapping: HP-SL230-4PORT ilo-ip: 172.30.0.48 ilo-password: "bananas" ilo-user: Administrator mac-addr: 8c:dc:d4:b5:c9:00 - id: compute2 ip-addr: 172.16.60.14 role: COMPUTE-ROLE server-group: RACK2 nic-mapping: HP-SL230-4PORT ilo-ip: 172.30.0.49 ilo-password: "bananas" ilo-user: Administrator mac-addr: 8c:dc:d4:b5:ce:80 # Ceph OSD Nodes - id: osd1 ip-addr: 172.16.60.15 role: OSD-ROLE server-group: RACK1 nic-mapping: HP-DL360-8PORT ilo-ip: 172.30.0.86 ilo-password: "bananas" ilo-user: Administrator mac-addr: 5c:b9:01:8d:6b:68 - id: osd2 ip-addr: 172.16.60.16 role: OSD-ROLE server-group: RACK2 nic-mapping: HP-DL360-8PORT ilo-ip: 172.30.0.87 ilo-password: "bananas" ilo-user: Administrator mac-addr: 5c:b9:01:8d:70:0c - id: osd3 ip-addr: 172.16.60.17 role: OSD-ROLE server-group: RACK3 nic-mapping: HP-DL360-8PORT ilo-ip: 172.30.0.88 ilo-password: "bananas" ilo-user: Administrator mac-addr: 5c:b9:01:8d:73:dc
server_roles.yml
This file links the servers in the servers.yml file with both the network interface and disk layout files. When using a separate deployer node it’s role would be defined here.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 server-roles: - name: CONTROLLER-ROLE interface-model: CONTROLLER-INTERFACES disk-model: CONTROLLER-DISKS - name: COMPUTE-ROLE interface-model: COMPUTE-INTERFACES disk-model: COMPUTE-DISKS - name: OSD-ROLE interface-model: OSD-INTERFACES disk-model: OSD-DISKS
server_groups.yml
Maps the physical layout of the servers to availability zones. It includes a reference to all the networks.
NOTE: This is where I’ve made the first significant change to the example model – I’ve added a reference to the new network that we’re adding called OSD-NET.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 server-groups: # # Server Groups provide a mechanism for organizing servers # into a hierarchy that reflected the physical topology. # # When allocating a server the configuration processor # will search down the hierarchy from the list of server # groups identified as the failure-zones for the control # plane until it finds an available server of the requested # role. If the allocation policy is "strict" servers are # allocated from different failure-zones. # # When determining which network from a network group to # associate with a server the configuration processor will # search up the hierarchy from the server group containing the # server until it finds a network in the required network # group. # # # In this example there is only one network in each network # group and so we put all networks in the top level server # group. Below this we create server groups for three # failure zones, within which servers are grouped by racks. # # Note: the association of servers to server groups is part # of the server definition (servers.yml) # # # At the top of the tree we have a server groups for # networks that can reach all servers # - name: CLOUD server-groups: - AZ1 - AZ2 - AZ3 networks: - EXTERNAL-API-NET - EXTERNAL-VM-NET - GUEST-NET - MANAGEMENT-NET - OSD-NET # # Create a group for each failure zone # - name: AZ1 server-groups: - RACK1 - name: AZ2 server-groups: - RACK2 - name: AZ3 server-groups: - RACK3 # # Create a group for each rack # - name: RACK1 - name: RACK2 - name: RACK3
nic_mappings.yml
As highlighted earlier this file contains all the “lspci” hardware network interface mappings on the various hardware server models used. In this deployment I’m using HP SL230s for the controller and compute nodes and HP DL360s for the OSD nodes.
Unfortunately there’s a little chicken and egg scenario here – in order to get the details for this file to complete the cloud the servers need an operating system (OS) on them, in order get an OS on the servers we need this file completed. The solution is to return to the installation media and temporarily drop (install) hLinux onto EACH MODEL of server that is being used. Then simply login and issue the following command capturing the output for your file:
sudo dmidecode | grep -A3 '^System Information'
lspci | grep -i eth
[Note: I have not included an example for the DL360 but this must be repeated for all different models and configurations used. For example if you have DL360s with different types, or numbers of interfaces then each server will need a dedicated profile.]
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 # nic-mappings are used to ensure that the device name used by the # operating system always maps to the same physical device. # A nic-mapping is associated to a server in the server definition. # The logical-name specified here can be used as a device name in # the network interface-models definitions. # # - name user-defined name for each mapping # physical-ports list of ports for this mapping # - logical-name device name to be used by the operating system # type physical port type # bus-address bus address of the physical device # # Notes: # - The PCI bus addresses are examples. You will need to determine # the values pertinent to your servers. These can be found with the # the `lspci` command or from the server BIOS # - enclose the bus address in quotation marks so yaml does not # misinterpret the embedded colon (:) characters # - simple-port is the only currently supported port type # - choosing a new device name prefix (e.g. 'eth' -> 'hed') will # help prevent remapping errors nic-mappings: - name: HP-SL230-4PORT physical-ports: - logical-name: hed0 type: simple-port bus-address: "0000:07:00.0" - logical-name: hed1 type: simple-port bus-address: "0000:07:00.1" - logical-name: hed2 type: simple-port bus-address: "0000:02:00.0" - logical-name: hed3 type: simple-port bus-address: "0000:02:00.1" - name: HP-DL360-2PORT physical-ports: - logical-name: hed0 type: simple-port bus-address: "0000:04:00.0" - logical-name: hed1 type: simple-port bus-address: "0000:04:00.1"
Server Disk Layouts
The following files have their disk-models->name referenced to the server_roles.yml file. The link is not to the file names themselves but to the content within the files. In theory you could give these files any valid yml filename once they are placed in the correct directory. Ansible simply reads all the files in the directory and generates a pool of key value pairs. Having just typed that I’m guessing that the disk-model names must be unique across all files used otherwise debugging will become very interesting.
disks_controller.yml
The controller nodes have 3 logical drives, 1 is used for the OS /dev/sda and the remaining 2 are used for Swift rings /dev/sdb & /dev/sdc.
[Note: In lab and test scenarios it is possible to use a single physical drive and use lvm mount points to provide the logical drives – performance may be challenging.]
My controllers all have 3 physical drives that will be presented as 3 logical drives – so I can use the default settings here.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 disk-models: - name: CONTROLLER-DISKS # This example is based on using a single 500GB disk for a volume # group that contains all file systems on a controller with 64GB # of memory. # # Additional disks can be added to the 'physical-volumes' section. # # If the available capacity of your servers is more that 500GB you # should consider using the "CONTROLLER-1TB-DISKS" disk-model # in disks_controller_1TB.yml instead. To use this alternative model # you need to edit the CONTROLLER-ROLE sections of server_roles.yml # volume-groups: - name: hlm-vg physical-volumes: # NOTE: 'sda_root' is a templated value. This value is checked in # os-config and replaced by the partition actually used on sda #e.g. sda1 or sda5 - /dev/sda_root # Add any additional disks for the volume group here # -/dev/sdx # -/dev/sdy logical-volumes: # The policy is not to consume 100% of the space of each volume group. # 5% should be left free for snapshots and to allow for some flexibility. - name: root size: 10% fstype: ext4 mount: / # Reserved space for kernel crash dumps # Should evaluate to a value that is slightly larger that # the memory size of your server - name: crash size: 13% mount: /var/crash fstype: ext4 mkfs-opts: -O large_file # Local Log files. - name: log size: 13% mount: /var/log fstype: ext4 mkfs-opts: -O large_file # Mysql Database. All persistent state from OpenStack services # is saved here. Although the individual objects are small the # accumulated data can grow over time - name: mysql size: 10% mount: /var/lib/mysql fstype: ext4 mkfs-opts: -O large_file consumer: name: mysql # Rabbitmq works mostly in memory, but needs to be able to persist # messages to disc under high load. This area should evaluate to a value # that is slightly larger than the memory size of your server - name: rabbitmq size: 13% mount: /var/lib/rabbitmq fstype: ext4 mkfs-opts: -O large_file consumer: name: rabbitmq rabbitmq_env: home # Database storage for event monitoring (Monasca). Events are generally # small data objects. - name: vertica size: 3% mount: /var/vertica fstype: ext4 mkfs-opts: -O large_file consumer: name: vertica # Messaging system for monitoring. - name: kafka size: 2% mount: /var/kafka fstype: ext4 mkfs-opts: -O large_file consumer: name: kafka # Data storage for centralized logging. This holds log entries from all # servers in the cloud and hence can require a lot of disk space. - name: elasticsearch size: 30% mount: /var/lib/elasticsearch fstype: ext4 # Zookeeper is used to provide cluster co-ordination in the monitoring # system. Although not a high user of disc space we have seen issues # with zookeeper snapshots filling up filesystems so we keep it in its # own space for stability. - name: zookeeper size: 1% mount: /var/lib/zookeeper fstype: ext4 consumer: name: os # Additional disk group defined for Swift device-groups: - name: swiftobj devices: - name: /dev/sdb - name: /dev/sdc # Add any additional disks for swift here # -name: /dev/sdd # -name: /dev/sde consumer: name: swift attrs: rings: - account - container - object-0
[Bonus Material : The following controller file is just for comparison of what the layout would look like when you squeeze everything onto a single drive controller]
Single Drive Controller Disk Layout Example [Not for Production]
Notice how the swift drives are now on a logical volume.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 disk-models: - name: CONTROLLER-1TB-DISKS # This example is based on using a single 1TB disk for a volume # group that contains all file systems on a controller with 64GB # of memory. # # Additional disks can be added to the 'physical-volumes' section. # # volume-groups: - name: hlm-vg physical-volumes: # NOTE: 'sda_root' is a templated value. This value is checked in # os-config and replaced by the partition actually used on sda #e.g. sda1 or sda5 - /dev/sda_root # Add any additional disks for the volume group here # -/dev/sdx # -/dev/sdy logical-volumes: # The policy is not to consume 100% of the space of each volume group. # 5% should be left free for snapshots and to allow for some flexibility. - name: root size: 5% fstype: ext4 mount: / # Reserved space for kernel crash dumps # Should evaluate to a value that is slightly larger that # the memory size of your server - name: crash size: 8% mount: /var/crash fstype: ext4 mkfs-opts: -O large_file # Local Log files. - name: log size: 10% mount: /var/log fstype: ext4 mkfs-opts: -O large_file # Mysql Database. All persistent state from OpenStack services # is saved here. Although the individual objects are small the # accumulated data can grow over time - name: mysql size: 8% mount: /var/lib/mysql fstype: ext4 mkfs-opts: -O large_file consumer: name: mysql # Rabbitmq works mostly in memory, but needs to be able to persist # messages to disc under high load. This area should evaluate to a value # that is slightly larger than the memory size of your server - name: rabbitmq size: 8% mount: /var/lib/rabbitmq fstype: ext4 mkfs-opts: -O large_file consumer: name: rabbitmq rabbitmq_env: home # Database storage for event monitoring (Monasca). Events are generally # small data objects. - name: vertica size: 5% mount: /var/vertica fstype: ext4 mkfs-opts: -O large_file consumer: name: vertica # Messaging system for monitoring. - name: kafka size: 2% mount: /var/kafka fstype: ext4 mkfs-opts: -O large_file consumer: name: kafka # Data storage for centralized logging. This holds log entries from all # servers in the cloud and hence can require a lot of disk space. - name: elasticsearch size: 20% mount: /var/lib/elasticsearch fstype: ext4 # Zookeeper is used to provide cluster co-ordination in the monitoring # system. Although not a high user of disc space we have seen issues # with zookeeper snapshots filling up filesystems so we keep it in its # own space for stability. - name: zookeeper size: 1% mount: /var/lib/zookeeper fstype: ext4 - name: LV_SWFPAC size: 25% fstype: xfs consumer: name: swift attrs: rings: - name: account - name: container - name: object-0 consumer: name: os
disks_compute.yml
The example configuration file expects two physical drives /dev/sda and /dev/sdb which is shown below
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 disk-models: - name: COMPUTE-DISKS # Disk model to be used for compute nodes # /dev/sda_root is used as a volume group for /, /var/log and /var/crash # sda_root is a templated value to align with whatever partition is really used # This value is checked in os config and replaced by the partition actually used # on sda e.g. sda1 or sda5 # /dev/sdb is used as a volume group for /var/lib (for VM storage) # Additional discs can be added to either volume group volume-groups: - name: hlm-vg physical-volumes: - /dev/sda_root logical-volumes: # The policy is not to consume 100% of the space of each volume group. # 5% should be left free for snapshots and to allow for some flexibility. - name: root size: 35% fstype: ext4 mount: / - name: log size: 50% mount: /var/log fstype: ext4 mkfs-opts: -O large_file - name: crash size: 10% mount: /var/crash fstype: ext4 mkfs-opts: -O large_file - name: vg-comp # this VG is dedicated to Nova Compute to keep VM IOPS off the OS disk physical-volumes: - /dev/sdb logical-volumes: - name: compute size: 95% mount: /var/lib/nova fstype: ext4 mkfs-opts: -O large_file
However, my lab only has single disk compute nodes so I’ve had to change the layout to match as follows
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 disk-models: - name: COMPUTE-DISKS # Disk model to be used for compute nodes # /dev/sda_root is used as a volume group for /, /var/log and /var/crash # sda_root is a templated value to align with whatever partition is really used # This value is checked in os config and replaced by the partition actually used # on sda e.g. sda1 or sda5 # /dev/sdb is used as a volume group for /var/lib (for VM storage) # Additional discs can be added to either volume group volume-groups: - name: hlm-vg physical-volumes: - /dev/sda_root logical-volumes: # The policy is not to consume 100% of the space of each volume group. # 5% should be left free for snapshots and to allow for some flexibility. - name: root size: 35% fstype: ext4 mount: / - name: log size: 20% mount: /var/log fstype: ext4 mkfs-opts: -O large_file - name: crash size: 10% mount: /var/crash fstype: ext4 mkfs-opts: -O large_file - name: compute size: 30% mount: /var/lib/nova fstype: ext4 mkfs-opts: -O large_file
disks_osd.yml
There are 8 physical drives available in the OSD servers. None of these drives are SSDs so very little point in journalling as I understand it. The most optimal configuration would be as follows using just a single drive for the OS and the remaining drives as OSDs:
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 disk-models: - name: OSD-DISKS # Disk model to be used for Ceph OSD nodes # /dev/sda_root is used as a volume group for /, /var/log and /var/crash # sda_root is a templated value to align with whatever partition is really used # This value is checked in os config and replaced by the partition actually used # on sda e.g. sda1 or sda5 volume-groups: - name: hlm-vg physical-volumes: - /dev/sda_root logical-volumes: # The policy is not to consume 100% of the space of each volume group. # 5% should be left free for snapshots and to allow for some flexibility. - name: root size: 30% fstype: ext4 mount: / - name: log size: 45% mount: /var/log fstype: ext4 mkfs-opts: -O large_file - name: crash size: 20% mount: /var/crash fstype: ext4 mkfs-opts: -O large_file consumer: name: os # Disks to be used by Ceph # Additional disks can be added if available device-groups: - name: ceph-osd-data0 devices: - name: /dev/sdb consumer: name: ceph attrs: usage: data - name: ceph-osd-data1 devices: - name: /dev/sdc consumer: name: ceph attrs: usage: data - name: ceph-osd-data2 devices: - name: /dev/sdd consumer: name: ceph attrs: usage: data - name: ceph-osd-data3 devices: - name: /dev/sde consumer: name: ceph attrs: usage: data - name: ceph-osd-data4 devices: - name: /dev/sdf consumer: name: ceph attrs: usage: data - name: ceph-osd-data5 devices: - name: /dev/sdg consumer: name: ceph attrs: usage: data - name: ceph-osd-data6 devices: - name: /dev/sdh consumer: name: ceph attrs: usage: data
Once again, despite a lack of high performant SSDs I’d like to test the mechanics of the Ceph journaling capability so I’ll use one standard drive as a journal. This gives a ratio of journal to osd 1:6 (this ratio is quite meaningless) .The configuration now becomes
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 disk-models: - name: OSD-DISKS # Disk model to be used for Ceph OSD nodes # /dev/sda_root is used as a volume group for /, /var/log and /var/crash # sda_root is a templated value to align with whatever partition is really used # This value is checked in os config and replaced by the partition actually used # on sda e.g. sda1 or sda5 volume-groups: - name: hlm-vg physical-volumes: - /dev/sda_root logical-volumes: # The policy is not to consume 100% of the space of each volume group. # 5% should be left free for snapshots and to allow for some flexibility. - name: root size: 30% fstype: ext4 mount: / - name: log size: 45% mount: /var/log fstype: ext4 mkfs-opts: -O large_file - name: crash size: 20% mount: /var/crash fstype: ext4 mkfs-opts: -O large_file consumer: name: os # Disks to be used by Ceph # Additional disks can be added if available device-groups: - name: ceph-osd-data-and-shared-journal-set-1 devices: - name: /dev/sdc consumer: name: ceph attrs: usage: data journal_disk: /dev/sdb - name: ceph-osd-data-and-shared-journal-set-2 devices: - name: /dev/sdd consumer: name: ceph attrs: usage: data journal_disk: /dev/sdb - name: ceph-osd-data-and-shared-journal-set-3 devices: - name: /dev/sde consumer: name: ceph attrs: usage: data journal_disk: /dev/sdb - name: ceph-osd-data-and-shared-journal-set-4 devices: - name: /dev/sdf consumer: name: ceph attrs: usage: data journal_disk: /dev/sdb - name: ceph-osd-data-and-shared-journal-set-5 devices: - name: /dev/sdg consumer: name: ceph attrs: usage: data journal_disk: /dev/sdb - name: ceph-osd-data-and-shared-journal-set-6 devices: - name: /dev/sdh consumer: name: ceph attrs: usage: data journal_disk: /dev/sdb
With the server disk layouts complete, that only leaves the network configuration files between us and the actual deployment start.
Here’s the network layout again to save paging back and forth (zoom in to see more detail on the servers)
net_interfaces.yml
Here we need to remember to add the new OSD replication network details. Once again for clarity, in a medium to large Ceph production deployment the OSD replication network would be on its own physical network interface cards.
I don’t have sufficient cabling and hardware to do this so I’ll trunk it up bond0 for demonstration purposes only.
Modify all the device names to match the nic mappings captured earlier.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 interface-models: # These examples uses hed3 and hed4 as a bonded # pair for all networks on all three server roles # # Edit the device names and bond options # to match your environment # - name: CONTROLLER-INTERFACES network-interfaces: - name: BOND0 device: name: bond0 bond-data: options: mode: active-backup miimon: 200 primary: hed0 provider: linux devices: - name: hed0 - name: hed1 network-groups: - EXTERNAL-API - EXTERNAL-VM - GUEST - MANAGEMENT - name: COMPUTE-INTERFACES network-interfaces: - name: BOND0 device: name: bond0 bond-data: options: mode: active-backup miimon: 200 primary: hed0 provider: linux devices: - name: hed0 - name: hed1 network-groups: - EXTERNAL-VM - GUEST - MANAGEMENT - name: OSD-INTERFACES network-interfaces: - name: BOND0 device: name: bond0 bond-data: options: mode: active-backup miimon: 200 primary: hed0 provider: linux devices: - name: hed0 - name: hed1 network-groups: - MANAGEMENT - OSD
Bonding Modes Reference for the curious
networks.yml
Configure this file to match the network settings in your parameter.
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 networks: # # This example uses the following networks # # Network CIDR VLAN # ------- ---- ---- # External API 10.0.1.0/24 101 (tagged) # External VM see note 1 102 (tagged) # Guest 10.1.1.0/24 103 (tagged) # Management 192.168.10.0/24 100 (untagged) # # Notes: # 1. Defined as part of Neutron configuration # # Modify these values to match your environment # - name: EXTERNAL-API-NET vlanid: 61 tagged-vlan: true cidr: 172.16.61.0/24 gateway-ip: 172.16.61.1 network-group: EXTERNAL-API - name: EXTERNAL-VM-NET vlanid: 62 tagged-vlan: true network-group: EXTERNAL-VM - name: GUEST-NET vlanid: 63 tagged-vlan: true cidr: 172.16.63.0/24 gateway-ip: 172.16.63.1 network-group: GUEST - name: MANAGEMENT-NET vlanid: 60 tagged-vlan: false cidr: 172.16.60.0/24 gateway-ip: 172.16.60.1 network-group: MANAGEMENT - name: OSD-NET vlanid: 64 tagged-vlan: true cidr: 172.16.64.0/24 gateway-ip: 172.16.64.1 network-group: OSD
Now we need to amend the network groups file to include the new OSD network
network_groups.yml
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 network-groups: # # External API # # This is the network group that users will use to # access the public API endpoints of your cloud # - name: EXTERNAL-API hostname-suffix: extapi load-balancers: - provider: ip-cluster name: extlb # If external-name is set then public urls in keystone # will use this name instead of the IP address #You must either set this to a name that can be resolved in your network # or comment out this line to use IP addresses external-name: roles: - public tls-components: - default cert-file: my-public-cert # This is the name of the certificate that will be used on load balancer. # Replace this with name of file in "~helion/my_cloud/config/tls/certs/". # This is the certificate that matches your setting for external-name # # Note that it is also possible to have per service certificates: # # cert-file: # default: my-public-cert # horizon: my-horizon-cert # nova-api: my-nova-cert # # # External VM # # This is the network group that will be used to provide # external access to VMs (via floating IP Addresses) # - name: EXTERNAL-VM tags: - neutron.l3_agent.external_network_bridge # # GUEST # # This is the network group that will be used to provide # private networks to VMs # - name: GUEST hostname-suffix: guest tags: - neutron.networks.vxlan # # Management # # This is the network group that will be used to for # management traffic within the cloud. # # The interface used by this group will be presented # to Neutron as physnet1, and used by provider VLANS # - name: MANAGEMENT hostname-suffix: mgmt hostname: true component-endpoints: - default routes: - default load-balancers: - provider: ip-cluster name: lb components: - default roles: - internal - admin tags: - neutron.networks.vlan: provider-physical-network: physnet1 # # OSD # # # This network group will be used to carry the to carry the internal Ceph # cluster replication traffic # - name: OSD hostanme-suffix: osd component-endpoints: - ceph-osd-internal
Whoops, sorry, did I say we were nearly finished – I forgot to mention the Ceph basic configurations.
As a minimum for the Ceph configuration it’s recommended to add a unique cluster id to the FSID parameter in ~/helion/my_cloud/config/ceph/settings.yml
The defaults will work however for a lab deployment.
Note: The online documentation recommends using “uuidgen” to create a new unique id but this application needs to be installed first.
sudo apt-get install uuid-runtime uuidgen
I’ve used 5e23fd0c-b449-43e3-a34e-fa0952c30e81 for the cluster FSID
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- # Variables specific to all ceph services fsid: 5e23fd0c-b449-43e3-a34e-fa0952c30e81 ceph_user: root ceph_group: root ceph_cluster: ceph ceph_release: firefly osd_pool_default_size: 3 osd_pool_default_pg_num: 128 # Variables specific to OSD fstype: xfs zap_data_disk: True persist_mountpoint: fstab osd_settle_time: 10 osd_journal_size: 5120 data_disk_poll_attempts: 5 data_disk_poll_interval: 12 # Variables specific to MON mon_default_dir: /var/lib/ceph/mon/{{ ceph_cluster }} # Additional parameters of ceph configuration can be injected via below section # extra: # section1: # key1: value # key2: value # section2: # key1: value # key2: value
Also for the Ceph client supply a unique secret_id in the ~/helion/my_cloud/config/ceph/user_model.yml
In my case I’ll use 7d4304a4-a4d2-4cea-ba3c-bbbdc8d11bd4 for the secret_id
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 ceph_user_models: - user: name: cinder type: openstack secret_id: 7d4304a4-a4d2-4cea-ba3c-bbbdc8d11bd4 pools: - name: volumes attrs: creation_policy: eager type: replicated replica_size: 3 permission: rwx pg: 100 usage: consumer: cinder-volume - name: vms attrs: creation_policy: eager type: replicated replica_size: 3 permission: rwx pg: 100 usage: consumer: nova - name: images attrs: creation_policy: lazy permission: rx - user: name: glance type: openstack pools: - name: images attrs: creation_policy: eager type: replicated replica_size: 3 permission: rwx pg: 128 usage: consumer: glance-datastore - user: name: cinder-backup type: openstack pools: - name: backups attrs: creation_policy: eager type: replicated replica_size: 3 permission: rwx pg: 128 usage: consumer: cinder-backup
Last, but not least, we have to add the new OSD network to the firewall
# # (c) Copyright 2015 Hewlett Packard Enterprise Development Company LP # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # --- product: version: 2 # # HOS will create firewall rules to enable the required access for # all of the deployed services. Use this section to define any # additional access. # # Each group of rules can be applied to one or more network groups # Examples are given for ping and ssh # # Names of rules, (e.g. "PING") are arbitrary and have no special significance # firewall-rules: # - name: SSH # # network-groups is a list of all the network group names # # that the rules apply to # network-groups: # - MANAGEMENT # rules: # - type: allow # # range of remote addresses in CIDR format that this # # rule applies to # remote-ip-prefix: 0.0.0.0/0 # port-range-min: 22 # port-range-max: 22 # # protocol must be one of: null, tcp, udp or icmp # protocol: tcp - name: PING network-groups: - MANAGEMENT - GUEST - EXTERNAL-API - OSD rules: # open ICMP echo request (ping) - type: allow remote-ip-prefix: 0.0.0.0/0 # icmp type port-range-min: 8 # icmp code port-range-max: 0 protocol: icmp
Okay, so there is one more file to be updated and in 3 weeks time you’ll be grateful you did it…any guesses, yes, that’s right, the ReadMe.MD file. We all love documentation, right? [Yes I do see the irony here]
README.MD ##Helion Entry Scale Cloud with Ceph and Separate OSD Replication Network## ## Updated - 15/01/16 ## Author - Grazzer ## Site - http://allthingscloud.eu ## Twitter - @allthingsclowd The input files in this example deploy a cloud that has the following characteristics: ### Control Planes ### - A single control plane consisting of three servers that co-host all of the required services. In addition, each hosts Ceph monitor service as well. ###Resource Pools### - Two compute Node - Three Ceph OSD Nodes *Additional resource nodes can be added to the configuration.* *Minimal Swift Resources are provided by the control plane* ###Deployer Node### This configuration runs the lifecycle-manager (formerly referred to as the deployer) on a control plane node. You need to include this node address in your servers.yml definition. This function does not need a dedicated network. *The minimum server count for this example is therefore 7 servers (Control Plane (x3) + Compute (x1) + OSD (x3))* An example set of servers are defined in ***data/servers.yml***. You will need to modify this file to reflect your specific environment. ###Networking### The example requires the following networks: IPMI/iLO network, connected to the deployer and the IPMI/iLO ports of all servers A pair of bonded NICs which are used by the following networks: - External API - This is the network that users will use to make requests to the cloud - External VM - This is the network that will be used to provide access to VMs (via floating IP addresses) - Guest - This is the network that will carry traffic between VMs on private networks within the cloud - Cloud Management - This is the network that will be used for all internal traffic between the cloud services, This network is also used to install and configure the nodes. This network needs to be on an untagged VLAN - OSD - This is the network that is used to carry all the Ceph OSD replication traffic Note that the EXTERNAL\_API network must be reachable from the EXTERNAL\_VM network if you want VMs to be able to make API calls to the cloud. An example set of networks are defined in ***data/networks.yml***. You will need to modify this file to reflect your environment. The example uses the devices hed3 & hed4 as a bonded network for all services. If you need to modify these for your environment they are defined in ***data/net_interfaces.yml*** The network devices eth3 & eth4 are renamed to devices hed4 & hed5 using the PCI bus mappings secified in ***data/nic_mappings.yml***. You may need to modify the PCI bus addresses to match your system. ###Local Storage### All servers should present a single OS disk, protected by a RAID controller. This disk needs to be at least 512GB in capacity. In addition the example configures one additional disk depending on the role of the server: - Controllers: /dev/sdb & /dev/sdc is configured to be used by Swift - Compute Severs: /dev/sda is configured as both OS and VM storage (not production grade) - OSD Servers: /dev/sdc, /dev/sdd, /dev/sde, /dev/sdf, /dev/sdg and /dev/sdh are configured for OSD data and /dev/sdb for journal storage Additional discs can be configured for any of these roles by editing the corresponding ***data/disks_*.yml*** file
We have now completed the configuration of our cloud model.
Note: IF GOING TO USE HELION DEVELOPMENT PLATFORM THEN A SERVICE NETWORK SHOULD ALSO BE ADDED NOW see here.
allthingscloud.eu 
One thought on “HOS 2.1 Ceph Installation with Network Customisation (4-of-8)”