SessionError: Queue dhcp_agent has been deleted

本帖最后由 Kevin517 于 2015-12-5 17:10 编辑

最近在搭建Openstack出了点问题，希望大家能帮忙解决。第一次发帖，不太懂规则，希望见谅。
一、环境：
两台CentOS6.5的服务器，双网卡

采用先电云计算平台，
controller节点部署模块：MySQL、Qpid、NTP、Keystone、Glance、Nova、Neutron、Dashboard、Swift、Heat、Ceilometer
compute节点部署模块：NTP、Nova、Neutron、Cinder、Ceilometer

controller：eth0:192.168.1.101；eth1：20.0.0.10
compute：eth0:192.168.1.102；eth1：20.0.0.20

内部网络为20.0.0.0/24

采用Flat网络模式，子网为20.0.0.110--20.0.0.200

网络拓扑

启用云主机正常，也可以正常使用。SSH正常。

二、详情

== Keystone users ==
+----------------------------------+------------+---------+------------------+
|             id             | name | enabled |    email    |
+----------------------------------+------------+---------+------------------+
| 41c6f7a07ec7416e9cc7c0616e49a256 | admin | True  |                |
| 380014868e1049e6992343273f4f9803 | ceilometer | True  | CEILOMETER_EMAIL |
| 5c83f9b71d8043a895ed082919e40c80 | cinder | True  | CINDER_EMAIL |
| b641a46511a04850ac875dc6a2ef02c8 | demo | True  | DEMO_EMAIL |
| 427e708806ae4ba2a09acbd26ae9a02e | glance | True  | GLANCE_EMAIL |
| ced5440eecc04e36bd4e8ef94b72f464 | heat | True  | HEAT_EMAIL |
| 8f18f4e04b1b4dac854ca0315da5067f |  neutron | True  |  NEUTRON_EMAIL |
| 6b531779a578477788d82be61aefc629 | nova | True  | NOVA_EMAIL |
| 157da08f4c334b52814d483b8f9f4cd4 | swift | True  | SWIFT_EMAIL |
+----------------------------------+------------+---------+------------------+
== Glance images ==
+--------------------------------------+-----------+-------------+------------------+-----------+--------+
| ID                                  | Name    | Disk Format | Container Format | Size    | Status |
+--------------------------------------+-----------+-------------+------------------+-----------+--------+
| 8613d83e-9016-4f41-b5a6-251367356f25 | centos6.5 | qcow2    | bare          | 305397760 | active |
+--------------------------------------+-----------+-------------+------------------+-----------+--------+
== Nova managed services ==
+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Binary          | Host    | Zone    | Status  | State | Updated_at                | Disabled Reason |
+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| nova-conductor | controller | internal | enabled | up | 2015-12-05T08:30:39.000000 | -             |
| nova-cert       | controller | internal | enabled | up | 2015-12-05T08:30:41.000000 | -             |
| nova-consoleauth | controller | internal | enabled | up | 2015-12-05T08:30:41.000000 | -             |
| nova-scheduler | controller | internal | enabled | up | 2015-12-05T08:30:41.000000 | -             |
| nova-compute    | compute | nova    | enabled | up | 2015-12-05T08:30:37.000000 | -             |
+------------------+------------+----------+---------+-------+----------------------------+-----------------+
== Nova networks ==
+--------------------------------------+------------+------+
| ID                                  | Label    | Cidr |
+--------------------------------------+------------+------+
| 053419db-c503-4be9-b23e-85db36ca0df3 | sharednet1 | - |
+--------------------------------------+------------+------+
== Nova instance flavors ==
+--------------------------------------+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID                                  | Name    | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+--------------------------------------+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1                                  | m1.tiny | 512    | 1 | 0       |    | 1    | 1.0       | True    |
| 2                                  | m1.small  | 2048    | 20 | 0       |    | 1    | 1.0       | True    |
| 3                                  | m1.medium | 4096    | 40 | 0       |    | 2    | 1.0       | True    |
| 4                                  | m1.large  | 8192    | 80 | 0       |    | 4    | 1.0       | True    |
| 4647ad72-cf5c-46e2-8141-9483d3926ec3 | paas    | 2048    | 20 | 10       | 4096 | 2    | 1.0       | True    |
| 5                                  | m1.xlarge | 16384    | 160  | 0       |    | 8    | 1.0       | True    |
+--------------------------------------+-----------+-----------+------+-----------+------+-------+-------------+-----------+
== Nova instances ==
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| ID                                  | Name | Status | Task State | Power State | Networks             |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| 1f25852d-664a-4244-9347-ee09efb5065f | broker  | ACTIVE | -       | Running    | sharednet1=20.0.0.110 |
| 47a18d56-8bc3-4586-8a3b-dd46928dfd2e | git    | ACTIVE | -       | Running    | sharednet1=20.0.0.114 |
| 9927027a-e03c-4a2c-bec1-e63c48cc4740 | node | ACTIVE | -       | Running    | sharednet1=20.0.0.111 |
| 6a287aad-9548-4218-91d0-d1ce4284aa36 | vmiaas2 | ACTIVE | -       | Running    | sharednet1=20.0.0.116 |
| 2e1bf07d-1a87-4864-bc57-9b6ebb9896ce | vmissa1 | ACTIVE | -       | Running    | sharednet1=20.0.0.115 |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+

=========================================================================

[root@controller ~]# neutron agent-list
+-------------------------------------------------+-------------------------+------------+---------+----------------+
| id                                                    | agent_type          | host          | alive | admin_state_up |
+-------------------------------------------------+-----------------------+--------------+---------+----------------+
| 2dde84be-c902-4cf6-ab80-983fa850751a  | DHCP agent          | controller    | :-)    | True          |
| 2fbc335c-6848-4f51-8702-a81f5676728f | L3 agent             | controller    | :-)    | True          |
| 3ca7b5d0-7137-4939-af97-025d05ec5b05 | Open vSwitch agent | controller    | :-)    | True          |
| 5ac65fa6-f5bd-418c-94dd-f349d6a50e9a | Open vSwitch agent | compute    | :-)    | True          |
| 84070851-205d-4804-8b8c-f78fb2dc1357 | Metadata agent    | compute    | :-)    | True          |
| c91fb6b0-e134-405f-bc5d-0fe2cf042692 | Metadata agent    | controller    | :-)    | True          |
| c9c5adb9-3898-4743-81ce-22d28cab13ff  | DHCP agent          | compute    | :-)    | True          |
+-------------------------------------------------+-----------------------+---------------+---------+----------------+

===========================================================================

[root@controller ~]# ovs-vsctl show
a0a800c7-2057-46cc-9deb-af9399717aae
Bridge br-int
      fail_mode: secure
      Port br-int
         Interface br-int
            type: internal
      Port int-br-ex
         Interface int-br-ex
Bridge br-ex
      Port phy-br-ex
         Interface phy-br-ex
      Port br-ex
         Interface br-ex
            type: internal
      Port "eth1"
         Interface "eth1"
ovs_version: "2.1.3"

============================================================================

[root@compute ~]# ovs-vsctl show
1ed084f4-6e73-4ef3-b5ee-6d07c38d72e5
Bridge br-int
      fail_mode: secure
      Port "qvo59be0229-2a"
         tag: 1
         Interface "qvo59be0229-2a"
      Port "tap54c100ca-45"
         tag: 1
         Interface "tap54c100ca-45"
            type: internal
      Port int-br-ex
         Interface int-br-ex
      Port "qvoa38d90be-af"
         tag: 1
         Interface "qvoa38d90be-af"
      Port "qvo462ecf20-03"
         tag: 1
         Interface "qvo462ecf20-03"
      Port "qvo81da423a-cf"
         tag: 1
         Interface "qvo81da423a-cf"
      Port br-int
         Interface br-int
            type: internal
      Port "qvoba05f9c4-21"
         tag: 1
         Interface "qvoba05f9c4-21"
Bridge br-ex
      Port br-ex
         Interface br-ex
            type: internal
      Port "eth1"
         Interface "eth1"
      Port phy-br-ex
         Interface phy-br-ex
ovs_version: "2.1.3"

===========================================================================

三、问题描述

在做完Neutron之后，先是重启了Nova，然后重启了Neutron。
查看Nova日志没有错误输出，查看Neutron日志，发现 dhcp-agent.log 错误输出。其他agent没有错误日志

下面贴出错误日志

[root@controller neutron]# cat dhcp-agent.log
2015-12-04 01:59:58.928 4924 INFO neutron.common.config [-] Logging enabled!
2015-12-04 01:59:58.943 4924 WARNING neutron.agent.linux.dhcp [req-9f9659c9-31da-411f-8b8b-884737d49a8d None] FAILED VERSION REQUIREMENT FOR DNSMASQ. DHCP AGENT MAY NOT RUN CORRECTLY! Please ensure that its version is 2.59 or above!
2015-12-04 01:59:59.009 4924 INFO neutron.openstack.common.rpc.impl_qpid [req-2c218c98-900a-43d5-a53f-84c31162583d None] Connected to AMQP server on controller:5672
2015-12-04 01:59:59.012 4924 INFO neutron.openstack.common.rpc.impl_qpid [req-2c218c98-900a-43d5-a53f-84c31162583d None] Connected to AMQP server on controller:5672
2015-12-04 01:59:59.036 4924 INFO neutron.openstack.common.rpc.impl_qpid [req-2c218c98-900a-43d5-a53f-84c31162583d None] Connected to AMQP server on controller:5672
2015-12-04 01:59:59.045 4924 INFO neutron.agent.dhcp_agent [req-2c218c98-900a-43d5-a53f-84c31162583d None] DHCP agent started
2015-12-04 02:00:03.553 4924 INFO neutron.openstack.common.service [req-2c218c98-900a-43d5-a53f-84c31162583d None] Caught SIGTERM, exiting
2015-12-04 02:00:04.050 5199 INFO neutron.common.config [-] Logging enabled!
2015-12-04 02:00:04.065 5199 WARNING neutron.agent.linux.dhcp [req-ca0e8553-836b-4214-8cb3-48f71bda299c None] FAILED VERSION REQUIREMENT FOR DNSMASQ. DHCP AGENT MAY NOT RUN CORRECTLY! Please ensure that its version is 2.59 or above!
2015-12-04 02:00:04.127 5199 INFO neutron.openstack.common.rpc.impl_qpid [req-b62d713f-6314-423a-8dc5-c1a7479652f6 None] Connected to AMQP server on controller:5672
2015-12-04 02:00:04.130 5199 INFO neutron.openstack.common.rpc.impl_qpid [req-b62d713f-6314-423a-8dc5-c1a7479652f6 None] Connected to AMQP server on controller:5672
2015-12-04 02:00:04.199 5199 INFO neutron.openstack.common.rpc.impl_qpid [req-b62d713f-6314-423a-8dc5-c1a7479652f6 None] Connected to AMQP server on controller:5672
2015-12-04 02:00:04.208 5199 INFO neutron.agent.dhcp_agent [req-b62d713f-6314-423a-8dc5-c1a7479652f6 None] DHCP agent started
2015-12-04 02:00:04.574 5199 INFO neutron.agent.dhcp_agent [req-b62d713f-6314-423a-8dc5-c1a7479652f6 None] Synchronizing state
2015-12-04 02:00:04.628 5199 INFO neutron.agent.dhcp_agent [req-b62d713f-6314-423a-8dc5-c1a7479652f6 None] Synchronizing state complete
2015-12-04 02:01:12.191 5199 INFO neutron.openstack.common.service [-] Caught SIGTERM, exiting
2015-12-04 02:01:12.690 5607 INFO neutron.common.config [-] Logging enabled!
2015-12-04 02:01:12.704 5607 WARNING neutron.agent.linux.dhcp [req-5f509eef-9bd3-453c-8726-798ed06ded36 None] FAILED VERSION REQUIREMENT FOR DNSMASQ. DHCP AGENT MAY NOT RUN CORRECTLY! Please ensure that its version is 2.59 or above!
2015-12-04 02:01:12.766 5607 INFO neutron.openstack.common.rpc.impl_qpid [req-81407814-7504-4a61-a420-8fa3ae1e390b None] Connected to AMQP server on controller:5672
2015-12-04 02:01:12.768 5607 INFO neutron.openstack.common.rpc.impl_qpid [req-81407814-7504-4a61-a420-8fa3ae1e390b None] Connected to AMQP server on controller:5672
2015-12-04 02:01:12.792 5607 INFO neutron.openstack.common.rpc.impl_qpid [req-81407814-7504-4a61-a420-8fa3ae1e390b None] Connected to AMQP server on controller:5672
2015-12-04 02:01:12.800 5607 INFO neutron.agent.dhcp_agent [req-81407814-7504-4a61-a420-8fa3ae1e390b None] DHCP agent started
2015-12-04 02:01:13.199 5607 INFO neutron.agent.dhcp_agent [req-81407814-7504-4a61-a420-8fa3ae1e390b None] Synchronizing state
2015-12-04 02:01:13.254 5607 INFO neutron.agent.dhcp_agent [req-81407814-7504-4a61-a420-8fa3ae1e390b None] Synchronizing state complete
2015-12-04 02:06:17.026 5607 INFO neutron.openstack.common.service [-] Caught SIGTERM, exiting
2015-12-04 02:06:17.517 6329 INFO neutron.common.config [-] Logging enabled!
2015-12-04 02:06:17.532 6329 WARNING neutron.agent.linux.dhcp [req-973a473d-3f44-42ec-980c-365d91a58039 None] FAILED VERSION REQUIREMENT FOR DNSMASQ. DHCP AGENT MAY NOT RUN CORRECTLY! Please ensure that its version is 2.59 or above!
2015-12-04 02:06:17.595 6329 INFO neutron.openstack.common.rpc.impl_qpid [req-3484c35c-b111-4572-9a2c-114300c7a41f None] Connected to AMQP server on controller:5672
2015-12-04 02:06:17.598 6329 INFO neutron.openstack.common.rpc.impl_qpid [req-3484c35c-b111-4572-9a2c-114300c7a41f None] Connected to AMQP server on controller:5672
2015-12-04 02:06:17.677 6329 INFO neutron.openstack.common.rpc.impl_qpid [req-3484c35c-b111-4572-9a2c-114300c7a41f None] Connected to AMQP server on controller:5672
2015-12-04 02:06:17.686 6329 INFO neutron.agent.dhcp_agent [req-3484c35c-b111-4572-9a2c-114300c7a41f None] DHCP agent started
2015-12-04 02:06:18.037 6329 INFO neutron.agent.dhcp_agent [req-3484c35c-b111-4572-9a2c-114300c7a41f None] Synchronizing state
2015-12-04 02:06:18.092 6329 INFO neutron.agent.dhcp_agent [req-3484c35c-b111-4572-9a2c-114300c7a41f None] Synchronizing state complete
2015-12-04 03:00:47.371 6329 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: Queue dhcp_agent has been deleted. (qpid/broker/Queue.cpp:1855)(408)
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid    return method(*args, **kwargs)
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid    nxt_receiver = self.session.next_receiver(timeout=timeout)
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid    if self._ecwait(lambda: self.incoming, timeout):
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid    result = self._ewait(lambda: self.closed or predicate(), timeout)
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 572, in _ewait
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid    self.check_error()
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 561, in check_error
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid    raise self.error
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid SessionError: Queue dhcp_agent has been deleted. (qpid/broker/Queue.cpp:1855)(408)
2015-12-04 03:00:47.371 6329 TRACE neutron.openstack.common.rpc.impl_qpid
2015-12-04 03:00:47.378 6329 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on controller:5672
2015-12-05 03:18:05.763 6329 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: Queue dhcp_agent has been deleted. (qpid/broker/Queue.cpp:1855)(408)
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 541, in ensure
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid    return method(*args, **kwargs)
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 598, in _consume
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid    nxt_receiver = self.session.next_receiver(timeout=timeout)
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "<string>", line 6, in next_receiver
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid    if self._ecwait(lambda: self.incoming, timeout):
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid    result = self._ewait(lambda: self.closed or predicate(), timeout)
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 572, in _ewait
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid    self.check_error()
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 561, in check_error
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid    raise self.error
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid SessionError: Queue dhcp_agent has been deleted. (qpid/broker/Queue.cpp:1855)(408)
2015-12-05 03:18:05.763 6329 TRACE neutron.openstack.common.rpc.impl_qpid
2015-12-05 03:18:05.770 6329 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on controller:5672

出现 SessionError: Queue dhcp_agent has been deleted. 这个错误。做了好多次，都是在这里出问题。

在启用云主机后，控制台加载显示 ”无法解析服务器的 DNS 地址。“

控制台.png

日志可以看得到

============================================================================

现在就想解决网络问题，因为后面的监控服务看不到有数据输出，再后面我配置 Sahara 使用Neutron为云网络的时候Dashboard直接进不去。不知道是不是跟这个问题有关系。

我对Open vSwitch 了解的不太多，没有使用GRE网络。

希望大神帮我看看。顺便指点一下这种错误的分析方法。