, , ,

By

Graham Land

·

Helion OpenStack 2.X Nova Instance Resize – Enable/Disable

By design Helion OpenStack 2.X has Nova Instance Resize/Cold Migrate disabled due to security implications. This blog tells you haw to re-enable these capabilities.

Enable Instance Resize and Cold Migrate

By design, Helion OpenStack 2.X has Nova Instance Resize and Nova Cold Migration disabled as these services present a security risk the way they are currently implemented in OpenStack (Kilo).

The Security Risk

Both of these services require passwordless ssh access with the ability to modify the underlying filesystem. A compute node has to be able to access all other compute nodes. This results in no containment between hosts should an attacker or virus gain access to any one of them.

Future: There are several blueprints upstream trying to address this by leveraging the secure Nova Live Migration process with some modifications – this uses libvirt/qemu to transfer data securely via TLS.

If you are willing to accept the above Security Risk then proceed –

Procedure

This should be performed by a competent Linux/OpenStack engineer. The steps are quite straight forward:

Enable Instance Resize

  • Enable passwordless ssh on all compute nodes
  • Enable resize and cold migration in the nova policy.json file and redeploy the nova services using the standard HLM process.
  • If you wish to have this capability within the Horizon GUI also modify the nova_policy.json file on each Horizon controller node  and redeploy the horizon services using the standard HLM process.

For example:

# On the deployer node - generate a key pair (this only needs to be done once, the same key pair should be used on every compute host)

ssh-keygen -t rsa -f ~/nova_resize/id_rsa -N ''

#This will result in two files the private key (id_rsa) and the public key (id_rsa.pub)

# transfer the private key - repeat this for all compute nodes
scp nova_resize/id_rsa* stack@helion-cp1-comp0001-mgmt:~/

# On each compute host ensure the following folder exists:

mkdir /var/lib/nova/.ssh

# On each compute host copy the private key to the .ssh folder and set appropriate permissions/owner

cp id_rsa /var/lib/nova/.ssh
chmod 600 /var/lib/nova/.ssh/id_rsa
chown nova:nova /var/lib/nova/.ssh/id_rsa

# On each host add the public key to the list of authorized keys

cat id_rsa.pub >> /var/lib/nova/.ssh/authorized_keys

# Turn off strict host checking on each compute host

echo 'StrictHostKeyChecking no' >> /var/lib/nova/.ssh/config

# On each compute host ensure the nova user can login to a shell

usermod -s /bin/bash nova

# Modify the various policy.json files to allow migrate and resize

"compute:resize": "rule:admin_api",

"compute_extension:admin_actions:migrate": "rule:admin_api",

# On each controller node modify the Horizon nova_policy.json file to enable resize

"compute:resize": "",

"compute:confirm_resize": "",

"compute:revert_resize": "", 

# Restart the Nova and Horizon Services on all nodes.

# Both Nova Instance Resize and Nova Cold Migration should now be operational.

 

Disable Instance Resize

  • Simply reverse the steps outlined above.

Caution

One of the big challenges with making configuration changes like this ‘manually’ is that when you go to redeploy or upgrade your cloud these changes have not been captured in the Helion Lifecycle Manager’s configuration repository and therefore will not be there in a redeployed cloud.

Solution – Use Ansible Playbooks for all these configuration changes and add them to the HLM git repo

I’m new to Ansible myself but was able to quickly put together the following playbooks to implement the manual procedures identified above. It’s well worth investing a few days playing with Ansible if you haven’t done so before – it’s a fantastic productivity enabler where Helion OpenStack is concerned.

Please ensure you thoroughly test these scripts before implementing on a production system – as mentioned they’re my first delve into the tool.

Instructions to use the playbooks can be found in the comments located within each playbook.

HOS 2.X Enable Instance Resize Playbook

# Author: Graham Joseph Land
# Blog: allthingscloud.eu
# Twitter: @allthingsclowd
# Email: graham@the above domain
# Date: 03/03/2016
# 
# Name: nova-enable-resize.yml
# Version: 1.1
# Purpose: This playbook is one of 5 that must be run in order to re-enable nova instance resize and cold migration
#
# Instructions: 
# 1. Copy this playbook to the following two ansible directories on the deployer node -> ~/scratch/ansible/next/hos/ansible AND ~/helion/hos/ansible
# 2. Log on to the Helion Lifecycle Manager (HLM Deployer) as the deployer user and change to this directory -> ~/scratch/ansible/next/hos/ansible
# 3. Enable nova instance resize and nova cold-migration by performing the following steps:
#      a.) ansible-playbook -v -i hosts/verb_hosts nova-enable-resize.yml
#      b.) cd ~/helion/hos/ansible
#      c.) git add -A
#      d.) git commit -m "Reconfigured Nova's policy.json to allow instance resize and cold migration for admins"
#      e.) cd ~/helion/hos/ansible
#      f.) ansible-playbook -i hosts/localhost config-processor-run.yml
#      g.) cd ~/helion/hos/ansible
#      h.) ansible-playbook -i hosts/localhost ready-deployment.yml
#      i.) cd ~/scratch/ansible/next/hos/ansible
#      j.) ansible-playbook -i hosts/verb_hosts nova-reconfigure.yml
#       k.) ansible-playbook -i hosts/verb_hosts horizon-reconfigure.yml

---
#
# on the deployer node
# 
    - hosts: OPS-LM

      tasks:
# get username of the current user
      - name: get the current username running this playbook
        local_action: command whoami
        register: username_on_the_host

# create temporary directory
      - name: create directory to store ssh keys
        file: path=~/nova_resize state=directory

# if it's a re-run remove existing private key to facilitate new key creation      
      - name: for repeat runs remove old private key from directory
        file: path=~/nova_resize/id_rsa state=absent

# if it's a re-run remove existing public key to facilitate new key creation
      - name: for repeat runs remove old public key from directory
        file: path=~/nova_resize/id_rsa.pub state=absent

# create new ssh keys
      - name: creating SSH keys for nova user login
        command: ssh-keygen -t rsa -f '~/nova_resize/id_rsa' -N ""

# modify nova policy.json file in mycloud repo to enable resize
      - name: modify nova policy.json file in deployer git repo - enable resize
        lineinfile: 'dest=~/helion/my_cloud/config/nova/policy.json state=present regexp="compute:resize" create=yes line="    \"compute:resize\": \"rule:admin_api\"," owner={{ username_on_the_host.stdout }} group={{ username_on_the_host.stdout }} mode=0777'

# modify nova policy.json file in mycloud repo to enable cold migration
      - name: modify nova policy.json file in deployer git repo - enable cold migrate
        lineinfile: 'dest=~/helion/my_cloud/config/nova/policy.json state=present regexp="\"compute_extension:admin_actions:migrate\"" create=yes line="    \"compute_extension:admin_actions:migrate\": \"rule:admin_api\"," owner={{ username_on_the_host.stdout }} group={{ username_on_the_host.stdout }} mode=0777'

#
# on each compute node
#

    - hosts: NOV-CMP
      sudo: yes
      vars:
# load the public ssh key into a variable
        nova_public_ssh_key: "{{ lookup('file', '~/nova_resize/id_rsa.pub') }}"

      tasks:
# create a .ssh directory if it doesn't already exist
      - name: create nova ssh directory on compute host
        file: path=/var/lib/nova/.ssh state=directory owner=nova group=nova mode=0755

# copy the private key to the .ssh directory
      - name: copy nova ssh private key to .ssh directory and set correct permissions on compute nodes
        copy: src=/home/stack/nova_resize/id_rsa dest=/var/lib/nova/.ssh/id_rsa owner=nova group=nova mode=0600

# add the public ssh key to the authorized_hosts file
      - name:  add nova ssh public key to the authorized_keys file on each compute nodes
        lineinfile: 'dest=/var/lib/nova/.ssh/authorized_keys state=present backup=yes create=yes line="\"{{ nova_public_ssh_key }}\"" owner=nova group=nova mode=0644'

# set stricthostkeychecking to no in the ssh config file
      - name: turn off strict host checking on each compute nodes
        lineinfile: dest=/var/lib/nova/.ssh/config state=present backup=yes regexp="^StrictHostKeyChecking" create=yes line="StrictHostKeyChecking no" owner=nova group=nova mode=0644

# set bash shell as default for nova user
      - name: set nova user shell on each compute node
        command: usermod -s /bin/bash nova

# verify that nova user can login without a password locally
      - name: verify passwordless login for nova user
        become: yes
        become_user: nova
        command: ssh nova@{{ inventory_hostname }} 'echo Passwordless Login Successfully!'

#
# on each controller node
#
    - hosts: HZN-WEB
      sudo: yes

      tasks:

# find path to horizons nova_policy_json file
      - shell:  find /opt/stack/venv/ -name nova_policy.json | grep dashboard
        register: nova_policy_json

# enable nova instance resize in horizons nova_policy_json
      - name: enable nova instance resize in horizon
        lineinfile: 'dest={{ nova_policy_json.stdout }} state=present backup=yes regexp="compute:resize" create=yes line="    \"compute:resize\": \"rule:admin_api\"," owner=horizon-venv group=horizon-venv mode=0664'

# enable nova instance resize in horizons nova_policy_json
      - name: enable nova instance confirm resize in horizon
        lineinfile: 'dest={{ nova_policy_json.stdout }} state=present backup=yes regexp="compute:confirm_resize" create=yes line="    \"compute:confirm_resize\": \"rule:admin_api\"," owner=horizon-venv group=horizon-venv mode=0664'

# enable nova instance resize in horizons nova_policy_json
      - name: enable nova instance revert resize in horizon
        lineinfile: 'dest={{ nova_policy_json.stdout }} state=present backup=yes regexp="compute:revert_resize" create=yes line="    \"compute:revert_resize\": \"rule:admin_api\"," owner=horizon-venv group=horizon-venv mode=0664'

 

HOS 2.X Disable Instance Resize Playbook

# Author: Graham Joseph Land
# Blog: allthingscloud.eu
# Twitter: @allthingsclowd
# Email: graham@the above domain
# Date: 04/03/2016
# 
# Name: nova-disable-resize.yml
# Version: 1.1
# Purpose: This playbook is one of 5 that must be run in order to disable nova instance resize and cold migration
#
# Instructions: 
# 1. Copy this playbook to the following two ansible directories on the deployer node -> ~/scratch/ansible/next/hos/ansible AND ~/helion/hos/ansible
# 2. Log on to the Helion Lifecycle Manager (HLM Deployer) as the deployer user and change to this directory -> ~/scratch/ansible/next/hos/ansible
# 3. Disable nova instance resize and nova cold-migration by performing the following steps:
#      a.) ansible-playbook -v -i hosts/verb_hosts nova-disable-resize.yml
#      b.) cd ~/helion/hos/ansible
#      c.) git add -A
#      d.) git commit -m "Reconfigured Nova's policy.json to disable instance resize and cold migration for admins"
#      e.) cd ~/helion/hos/ansible
#      f.) ansible-playbook -i hosts/localhost config-processor-run.yml
#      g.) cd ~/helion/hos/ansible
#      h.) ansible-playbook -i hosts/localhost ready-deployment.yml
#      i.) cd ~/scratch/ansible/next/hos/ansible
#      j.) ansible-playbook -i hosts/verb_hosts nova-reconfigure.yml
#       k.) ansible-playbook -i hosts/verb_hosts horizon-reconfigure.yml

---
#
# on each controller node
#
    - hosts: HZN-WEB
      sudo: yes

      tasks:

# find path to horizons nova_policy_json file
      - shell:  find /opt/stack/venv/ -name nova_policy.json | grep dashboard
        register: nova_policy_json

# disable nova instance resize in horizons nova_policy_json
      - name: disable nova instance resize in horizon
        lineinfile: 'dest={{ nova_policy_json.stdout }} state=present backup=yes regexp="compute:resize" create=yes line="    \"compute:resize\": \"!\"," owner=horizon-venv group=horizon-venv mode=0664'

# disable nova instance resize in horizons nova_policy_json
      - name: disable nova instance confirm resize in horizon
        lineinfile: 'dest={{ nova_policy_json.stdout }} state=present backup=yes regexp="compute:confirm_resize" create=yes line="    \"compute:confirm_resize\": \"!\"," owner=horizon-venv group=horizon-venv mode=0664'

# disable nova instance resize in horizons nova_policy_json
      - name: disable nova instance revert resize in horizon
        lineinfile: 'dest={{ nova_policy_json.stdout }} state=present backup=yes regexp="compute:revert_resize" create=yes line="    \"compute:revert_resize\": \"!\"," owner=horizon-venv group=horizon-venv mode=0664'
#
# on each compute node
#

    - hosts: NOV-CMP
      sudo: yes

# load the public ssh key into a variable
      vars:
        nova_public_ssh_key: "{{ lookup('file', '~/nova_resize/id_rsa.pub') }}"
      
      tasks:
# remove ssh key from authorized_keys file 
      - authorized_key: 'user=nova key="{{ nova_public_ssh_key }}" path="/var/lib/nova/.ssh/authorized_keys" state=absent'

# remove the private key to the .ssh directory
      - name: copy nova ssh private key to .ssh directory and set correct permissions on compute nodes
        file: path=/var/lib/nova/.ssh/id_rsa state=absent

# set stricthostkeychecking to yes in the ssh config file
      - name: turn on strict host checking on each compute nodes
        lineinfile: dest=/var/lib/nova/.ssh/config state=present backup=yes regexp="^StrictHostKeyChecking" create=yes line="StrictHostKeyChecking yes" owner=nova group=nova mode=0644

# unset bash shell as default for nova user
      - name: unset nova user shell on each compute node
        command: usermod -s /bin/false nova

# verify that nova user can't login without a password locally
      - name: verify passwordless login fails for nova user - RED FAILURE IS GOOD!!!
        become: yes
        become_user: nova
        command: ssh nova@{{ inventory_hostname }} 'echo Passwordless Login Successfully ... NOT!'
        ignore_errors: yes

#
# on the deployer node
# 
    - hosts: OPS-LM
        
      tasks:
# get username of the current user
      - name: get the current username running this playbook
        local_action: command whoami
        register: username_on_the_host

# if it's a re-run remove existing private key to facilitate new key creation      
      - name: remove private ssh key
        file: path=~/nova_resize/id_rsa state=absent

# delete existing public key 
      - name: delete public ssh key
        file: path=~/nova_resize/id_rsa.pub state=absent

# modify nova policy.json file in mycloud repo to disable resize
      - name: modify nova policy.json file in deployer git repo - disable instance resize
        lineinfile: 'dest=~/helion/my_cloud/config/nova/policy.json state=present regexp="compute:resize" create=yes line="    \"compute:resize\": \"\"," owner={{ username_on_the_host.stdout }} group={{ username_on_the_host.stdout }} mode=0777'

# modify nova policy.json file in mycloud repo to disable cold migration
      - name: modify nova policy.json file in deployer git repo - disable cold migrate
        lineinfile: 'dest=~/helion/my_cloud/config/nova/policy.json state=present regexp="\"compute_extension:admin_actions:migrate\"" create=yes line="    \"compute_extension:admin_actions:migrate\": \"\"," owner={{ username_on_the_host.stdout }} group={{ username_on_the_host.stdout }} mode=0777'

# delete temporary directory
      - name: create directory to store ssh keys
        file: path=~/nova_resize state=absent

 

 

 

Tags:

Leave a comment