Today I was asked to investigate a “failing OpenStack network” where a client had been evaluating the product for the past month. As always, nothing was changed it just stopped working š
Another challenge with this engagement was that I did not have direct access to the system – I was in another country and was not granted remote access. So, I produced a Google document with a detailed list of commands and shared this with the OpenStack operator and we used this as the basis for our collaboration and investigation. The next two blog posts are basically the final document that was generated.
The initial complaint was that the external network didn’t work.
This is what the failure looked like:
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron router-gateway-set 8c8a0159-2b37-4fae-93e9-b302a9d59573 35510045-decf-491e-9990-87a3f77f0284
Request Failed: internal server error while processing your request.
Don’t forget that you can add the “–debug” switch to get more verbose output..
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron --debug router-gateway-set 8c8a0159-2b37-4fae-93e9-b302a9d59573 35510045-decf-491e-9990-87a3f77f0284
DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG: keystoneclient.session RESP: [200] content-length: 423 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.10 (Debian) connection: Keep-Alive date: Fri, 20 Nov 2015 16:27:30 GMT content-type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "https://10.254.25.35:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
DEBUG: neutronclient.neutron.v2_0.router.SetGatewayRouter run(Namespace(disable_snat=False, external_network_id=u'35510045-decf-491e-9990-87a3f77f0284', request_format='json', router_id=u'8c8a0159-2b37-4fae-93e9-b302a9d59573'))
DEBUG: keystoneclient.auth.identity.v2 Making authentication request to https://10.254.25.35:5000/v2.0/tokens
DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:9696/v2.0/routers.json?fields=id&id=8c8a0159-2b37-4fae-93e9-b302a9d59573 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a"
DEBUG: keystoneclient.session RESP: [200] date: Fri, 20 Nov 2015 16:27:31 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 61 x-openstack-request-id: req-b6469ccc-b81e-4ae1-82ea-e0413bd46f3d
RESP BODY: {"routers": [{"id": "8c8a0159-2b37-4fae-93e9-b302a9d59573"}]}
DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:9696/v2.0/networks.json?fields=id&id=35510045-decf-491e-9990-87a3f77f0284 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a"
DEBUG: keystoneclient.session RESP: [200] date: Fri, 20 Nov 2015 16:27:31 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 62 x-openstack-request-id: req-e39fba85-c775-4296-a52f-a83731a5b5f2
RESP BODY: {"networks": [{"id": "35510045-decf-491e-9990-87a3f77f0284"}]}
DEBUG: keystoneclient.session REQ: curl -i -X PUT https://10.254.25.35:9696/v2.0/routers/8c8a0159-2b37-4fae-93e9-b302a9d59573.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a" -d '{"router": {"external_gateway_info": {"network_id": "35510045-decf-491e-9990-87a3f77f0284"}}}'
DEBUG: keystoneclient.session RESP:
DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}
ERROR: neutronclient.shell Request Failed: internal server error while processing your request.
Traceback (most recent call last):
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/shell.py", line 701, in run_subcommand
return run_command(cmd, cmd_parser, sub_argv)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/shell.py", line 92, in run_command
return cmd.run(known_args)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/neutron/v2_0/router.py", line 205, in run
neutron_client.add_gateway_router(_router_id, router_dict)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
ret = self.function(instance, *args, **kwargs)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 432, in add_gateway_router
body={'router': {'external_gateway_info': body}})
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1322, in put
headers=headers, params=params)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1290, in retry_request
headers=headers, params=params)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1244, in do_request
self._handle_fault_response(status_code, replybody)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1218, in _handle_fault_response
exception_handler_v20(status_code, des_error_body)
File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 66, in exception_handler_v20
status_code=status_code)
InternalServerError: Request Failed: internal server error while processing your request.
Check the log files for the router that was created as follows:
grep 8c8a0159-2b37-4fae-93e9-b302a9d59573 -l -R /var/log/*
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# grep 8c8a0159-2b37-4fae-93e9-b302a9d59573 -l -R /var/log/*
/var/log/horizon/horizon-json.log
/var/log/horizon/horizon_error.log
/var/log/neutron/neutron-json.log
/var/log/neutron/neutron.log
Investigation of the neutron.log file has the root cause of the problem –
“TooManyExternalNetworks: More than one external network exists”
A quick check with neutron reveals the existence of two external networks – not supported by this version of OpenStack.
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-external-list
+--------------------------------------+---------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-----------------------------------------------------+
| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |
| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct | |
+--------------------------------------+---------+-----------------------------------------------------+
The solution is simple – remove the second external network.
Remove the new external network called direct.
List all the current networks
neutron net-list
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-list
+--------------------------------------+-------------+-------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------+-------------------------------------------------------+
| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |
| 4259ab26-a1c4-427e-a7e0-b941368c4e23 | default-net | d20f33e0-82f1-4ce1-80cf-17a27c86d11c 172.0.100.0/24 |
| 73ff3b91-77ad-4ef4-aa67-a99a5ef57650 | demo_net1 | 4a20b788-da40-48a0-b459-5034d2831adf 192.168.111.0/24 |
| a493f23f-3646-49c5-a342-82c0ac7601f6 | lab2-net| 75542425-b437-4433-a9d8-3009527e1435 192.168.1.0/24 |
| d2ad4daa-d004-40bd-afa8-b7e19b8dde67 | HPinternal | 28d80588-9bf1-415e-a1f3-b08d95d5bf14 10.0.0.0/24 |
| e70da3b0-ad8d-4ec0-9dd7-40ba563d2416 | SVC | 57929850-9c6a-4917-a8a4-61c033d72705 10.254.26.0/24 |
| e99f2cdd-eb42-4466-9b17-7cb85636c013 | lab1-net1 | 60b9efc0-081f-45fd-b06c-0ce88fc04d50 192.168.113.0/24 |
| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct | |
+--------------------------------------+-------------+-------------------------------------------------------+
Identify the external networks
neutron net-external-list
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-external-list
+--------------------------------------+---------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-----------------------------------------------------+
| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |
| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct | |
+--------------------------------------+---------+-----------------------------------------------------+
Delete the additional external network
neutron net-delete direct
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-delete direct
Deleted network: direct
root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-list
+--------------------------------------+-------------+-------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------+-------------------------------------------------------+
| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |
| 4259ab26-a1c4-427e-a7e0-b941368c4e23 | default-net | d20f33e0-82f1-4ce1-80cf-17a27c86d11c 172.0.100.0/24 |
| 73ff3b91-77ad-4ef4-aa67-a99a5ef57650 | demo_net1 | 4a20b788-da40-48a0-b459-5034d2831adf 192.168.111.0/24 |
| a493f23f-3646-49c5-a342-82c0ac7601f6 | lab2-net | 75542425-b437-4433-a9d8-3009527e1435 192.168.1.0/24 |
| d2ad4daa-d004-40bd-afa8-b7e19b8dde67 | HPinternal | 28d80588-9bf1-415e-a1f3-b08d95d5bf14 10.0.0.0/24 |
| e70da3b0-ad8d-4ec0-9dd7-40ba563d2416 | SVC | 57929850-9c6a-4917-a8a4-61c033d72705 10.254.26.0/24 |
| e99f2cdd-eb42-4466-9b17-7cb85636c013 | lab1-net1 | 60b9efc0-081f-45fd-b06c-0ce88fc04d50 192.168.113.0/24 |
+--------------------------------------+-------------+-------------------------------------------------------+
The next blog post will verify the health of both the Helion OpenStack basic services and the neutron network.
allthingscloud.eu 