Neutron : router-gateway-set to External Network Fails

Today I was asked to investigate a “failing OpenStack network” where a client had been evaluating the product for the past month. As always, nothing was changed it just stopped working 😉

Another challenge with this engagement was that I did not have direct access to the system – I was in another country and was not granted remote access. So, I produced a Google document with a detailed list of commands and shared this with the OpenStack operator and we used this as the basis for our collaboration and investigation. The next two blog posts are basically the final document that was generated.

The initial complaint was that the external network didn’t work.

This is what the failure looked like:

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron router-gateway-set 8c8a0159-2b37-4fae-93e9-b302a9d59573 35510045-decf-491e-9990-87a3f77f0284

Request Failed: internal server error while processing your request.

Don’t forget that you can add the “–debug” switch to get more verbose output..

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron --debug router-gateway-set 8c8a0159-2b37-4fae-93e9-b302a9d59573 35510045-decf-491e-9990-87a3f77f0284

DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"

DEBUG: keystoneclient.session RESP: [200] content-length: 423 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.10 (Debian) connection: Keep-Alive date: Fri, 20 Nov 2015 16:27:30 GMT content-type: application/json

RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "https://10.254.25.35:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}



DEBUG: neutronclient.neutron.v2_0.router.SetGatewayRouter run(Namespace(disable_snat=False, external_network_id=u'35510045-decf-491e-9990-87a3f77f0284', request_format='json', router_id=u'8c8a0159-2b37-4fae-93e9-b302a9d59573'))

DEBUG: keystoneclient.auth.identity.v2 Making authentication request to https://10.254.25.35:5000/v2.0/tokens

DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:9696/v2.0/routers.json?fields=id&id=8c8a0159-2b37-4fae-93e9-b302a9d59573 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a"

DEBUG: keystoneclient.session RESP: [200] date: Fri, 20 Nov 2015 16:27:31 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 61 x-openstack-request-id: req-b6469ccc-b81e-4ae1-82ea-e0413bd46f3d

RESP BODY: {"routers": [{"id": "8c8a0159-2b37-4fae-93e9-b302a9d59573"}]}



DEBUG: keystoneclient.session REQ: curl -i -X GET https://10.254.25.35:9696/v2.0/networks.json?fields=id&id=35510045-decf-491e-9990-87a3f77f0284 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a"

DEBUG: keystoneclient.session RESP: [200] date: Fri, 20 Nov 2015 16:27:31 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 62 x-openstack-request-id: req-e39fba85-c775-4296-a52f-a83731a5b5f2

RESP BODY: {"networks": [{"id": "35510045-decf-491e-9990-87a3f77f0284"}]}



DEBUG: keystoneclient.session REQ: curl -i -X PUT https://10.254.25.35:9696/v2.0/routers/8c8a0159-2b37-4fae-93e9-b302a9d59573.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}04e2fb3010f4069945373862d00e12d9e34c488a" -d '{"router": {"external_gateway_info": {"network_id": "35510045-decf-491e-9990-87a3f77f0284"}}}'

DEBUG: keystoneclient.session RESP:

DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Request Failed: internal server error while processing your request.", "type": "HTTPInternalServerError", "detail": ""}}

ERROR: neutronclient.shell Request Failed: internal server error while processing your request.

Traceback (most recent call last):

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/shell.py", line 701, in run_subcommand

   return run_command(cmd, cmd_parser, sub_argv)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/shell.py", line 92, in run_command

   return cmd.run(known_args)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/neutron/v2_0/router.py", line 205, in run

   neutron_client.add_gateway_router(_router_id, router_dict)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params

   ret = self.function(instance, *args, **kwargs)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 432, in add_gateway_router

   body={'router': {'external_gateway_info': body}})

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1322, in put

   headers=headers, params=params)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1290, in retry_request

   headers=headers, params=params)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1244, in do_request

   self._handle_fault_response(status_code, replybody)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1218, in _handle_fault_response

   exception_handler_v20(status_code, des_error_body)

 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 66, in exception_handler_v20

   status_code=status_code)

InternalServerError: Request Failed: internal server error while processing your request.

Check the log files for the router that was created as follows:

grep 8c8a0159-2b37-4fae-93e9-b302a9d59573 -l -R /var/log/*

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# grep 8c8a0159-2b37-4fae-93e9-b302a9d59573 -l -R /var/log/*

/var/log/horizon/horizon-json.log

/var/log/horizon/horizon_error.log

/var/log/neutron/neutron-json.log

/var/log/neutron/neutron.log

Investigation of the neutron.log file has the root cause of the problem –

“TooManyExternalNetworks: More than one external network exists”

 

neutron.log

A quick check with neutron reveals the existence of two external networks – not supported by this version of OpenStack.

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-external-list

+--------------------------------------+---------+-----------------------------------------------------+

| id                                    | name  | subnets                                               |

+--------------------------------------+---------+-----------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |

| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct  |                                                       |

+--------------------------------------+---------+-----------------------------------------------------+

The solution is simple – remove the second external network.

Remove the new external network called direct.

List all the current networks

neutron net-list

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-list

+--------------------------------------+-------------+-------------------------------------------------------+

| id                                    | name  | subnets                                                    |

+--------------------------------------+-------------+-------------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24   |

| 4259ab26-a1c4-427e-a7e0-b941368c4e23 | default-net | d20f33e0-82f1-4ce1-80cf-17a27c86d11c 172.0.100.0/24   |

| 73ff3b91-77ad-4ef4-aa67-a99a5ef57650 | demo_net1   | 4a20b788-da40-48a0-b459-5034d2831adf 192.168.111.0/24 |

| a493f23f-3646-49c5-a342-82c0ac7601f6 | lab2-net| 75542425-b437-4433-a9d8-3009527e1435 192.168.1.0/24   |

| d2ad4daa-d004-40bd-afa8-b7e19b8dde67 | HPinternal  | 28d80588-9bf1-415e-a1f3-b08d95d5bf14 10.0.0.0/24  |

| e70da3b0-ad8d-4ec0-9dd7-40ba563d2416 | SVC    | 57929850-9c6a-4917-a8a4-61c033d72705 10.254.26.0/24   |

| e99f2cdd-eb42-4466-9b17-7cb85636c013 | lab1-net1   | 60b9efc0-081f-45fd-b06c-0ce88fc04d50 192.168.113.0/24 |

| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct  |   |

+--------------------------------------+-------------+-------------------------------------------------------+

Identify the external networks

neutron net-external-list

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-external-list

+--------------------------------------+---------+-----------------------------------------------------+

| id                                    | name      | subnets |

+--------------------------------------+---------+-----------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24 |

| f917a3eb-957a-4c0a-baa8-8b20607440a0 | direct  |                                                      |

+--------------------------------------+---------+-----------------------------------------------------+

Delete the additional external network

neutron net-delete direct

root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-delete direct

Deleted network: direct



root@overcloud-ce-controller-controller0-dlmy4f5tbc5d:~# neutron net-list

+--------------------------------------+-------------+-------------------------------------------------------+

| id                                    | name          | subnets   |

+--------------------------------------+-------------+-------------------------------------------------------+

| 35510045-decf-491e-9990-87a3f77f0284 | ext-net    | bfaacf82-7a4f-4e0c-aab9-94f705dc2c11 10.254.27.0/24   |

| 4259ab26-a1c4-427e-a7e0-b941368c4e23 | default-net | d20f33e0-82f1-4ce1-80cf-17a27c86d11c 172.0.100.0/24   |

| 73ff3b91-77ad-4ef4-aa67-a99a5ef57650 | demo_net1   | 4a20b788-da40-48a0-b459-5034d2831adf 192.168.111.0/24 |

| a493f23f-3646-49c5-a342-82c0ac7601f6 | lab2-net   | 75542425-b437-4433-a9d8-3009527e1435 192.168.1.0/24   |

| d2ad4daa-d004-40bd-afa8-b7e19b8dde67 | HPinternal  | 28d80588-9bf1-415e-a1f3-b08d95d5bf14 10.0.0.0/24  |

| e70da3b0-ad8d-4ec0-9dd7-40ba563d2416 | SVC        | 57929850-9c6a-4917-a8a4-61c033d72705 10.254.26.0/24   |

| e99f2cdd-eb42-4466-9b17-7cb85636c013 | lab1-net1   | 60b9efc0-081f-45fd-b06c-0ce88fc04d50 192.168.113.0/24 |

+--------------------------------------+-------------+-------------------------------------------------------+

The next blog post will verify the health of both the Helion OpenStack basic services and the neutron network.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s