360 OpenStack支持IP SAN存储实现
背景:
为了更多的满足TOB场景下的需求，360虚拟化团队一直不断丰富完善openstack侧的功能，近期对接过信创存储腾凌存储等商业存储，所以梳理一下整个流程，下面进入正文从架构以及源码了解openstack如何支持SAN存储。
一、相关概念
提到 IP SAN 必然会想到磁盘阵列，磁盘阵列有三种架构分别为：DAS,NAS,SAN。而SAN里面主要又分为IP SAN和FC SAN。
FC-SAN（Fibre Channel Storage Area Network）是一种基于光纤通道技术的存储网络，它将存储设备和服务器连接在一起，形成一个高速、高性能的存储区域网络。FC-SAN的核心是光纤通道交换机，它实现了光纤通道协议，使得存储设备和服务器之间的数据传输更加可靠和高效。
IP-SAN（Internet Protocol Storage Area Network）是一种基于IP协议的存储网络，它将存储设备、连接设备和接口集成在高速网络中。IP-SAN使用IP网络将存储设备连接在一起，实现数据的可靠传输和共享。由于IP网络具有广泛的普及性和互操作性，IP-SAN具有较好的扩展性和灵活性。
DAS（Direct Attached Storage）是一种直接附加存储技术，它将存储设备通过电缆直接连接到服务器上。DAS的优点是简单、成本低，适用于小型网络和单机环境。但是，DAS的缺点也很明显，如存储容量受限、扩展性差、数据共享困难等。
NAS（Network Attached Storage）是一种网络附加存储技术，它将存储设备连接到现有的网络上，提供数据和文件服务。NAS实际上是一个专门优化了的文件服务器，具有独立的操作系统和文件系统。NAS的优点是易于部署和管理，可以实现数据的集中存储和共享。但是，NAS的缺点是性能受限于网络带宽和稳定性，不适合大规模数据存储和高性能计算。
二、OpenStack实现
2.1 cinder侧实现
我们知道在openstack中cinder负责存储volume 的生命周期管理，而cinder中cinder-volume负责转发控制面请求从而对存储执行 action，对于每一种存储介质，cinder-volume需要调用对应的driver才可以，这里以最近对接过的信创腾凌ip san 存储为例讲解实现。
我们首先需要配置一个cinder 的volume backend tldriver
[tldriver]
volume_backend_name = tldriver
volume_driver = cinder.volume.drivers.tengling.tengling_driver.TenglingISCSIDriver
tengling_sanip = {{ tengling_sanip }}
tengling_username = {{ tengling_username }}
tengling_password = {{ tengling_password }}
tengling_storagepool = {{ tengling_storagepool }}
tengling_max_clone_depth = {{ tengling_max_clone_depth }}
tengling_flatten_volume_from_snapshot=True我们以创建一个volume从源码刨析整个的流程，前面将新建的volume type指定capability 为backend tldriver，这样scheduler就能调度到我们的新存储上了。cinder scheduler 通过rpc 请求volume 调用 volumeManager 的create_volume函数
@objects.Volume.set_workers
def create_volume(self, context, volume, request_spec=None,
filter_properties=None, allow_reschedule=True):
..........
try:
# NOTE(flaper87): Driver initialization is
# verified by the task itself.
flow_engine = create_volume.get_flow(
context_elevated,
self,
self.db,
self.driver,
self.scheduler_rpcapi,
self.host,
volume,
allow_reschedule,
context,
request_spec,
filter_properties,
image_volume_cache=self.image_volume_cache,
)
except Exception:
msg = _("Create manager volume flow failed.")
LOG.exception(msg, resource={'type': 'volume', 'id': volume.id})
raise exception.CinderException(msg)cinder volume创建volume时通过task flow执行了核心任务 CreateVolumeFromSpecTask，这里用户创建了一个系统盘，指定了image，所以执行了_create_from_image ，最终调用了
_create_from_image_cache_or_download 方法
class CreateVolumeFromSpecTask(flow_utils.CinderTask):
..........
def execute(self, context, volume, volume_spec):
..........
elif create_type == 'image':
model_update = self._create_from_image(context,
volume,
**volume_spec)
..........
def _create_from_image(self, context, volume,
image_location, image_id, image_meta,
image_service, **kwargs):
..........
if not cloned:
model_update = self._create_from_image_cache_or_download(
context,
volume,
image_location,
image_id,
image_meta,
image_service)
def _create_from_image_cache_or_download(self, context, volume,
image_location, image_id,
image_meta, image_service,
update_cache=False):
..........
try:
if not cloned:
try:
with image_utils.TemporaryImages.fetch(
image_service, context, image_id,
backend_name) as tmp_image:
if CONF.verify_glance_signatures != 'disabled':
# Verify image signature via reading content from
# temp image, and store the verification flag if
# required.
verified = \
image_utils.verify_glance_image_signature(
context, image_service,
image_id, tmp_image)
self.db.volume_glance_metadata_bulk_create(
context, volume.id,
{'signature_verified': verified})
# Try to create the volume as the minimal size,
# then we can extend once the image has been
# downloaded.
data = image_utils.qemu_img_info(tmp_image)
virtual_size = image_utils.check_virtual_size(
data.virtual_size, volume.size, image_id)
if should_create_cache_entry:
if virtual_size and virtual_size != original_size:
volume.size = virtual_size
volume.save()
model_update = self._create_from_image_download(
context,
volume,
image_location,
image_meta,
image_service
)
finally:
# If we created the volume as the minimal size, extend it back to
# what was originally requested. If an exception has occurred or
# extending it back failed, we still need to put this back before
# letting it be raised further up the stack.
if volume.size != original_size:
try:
self.driver.extend_volume(volume, original_size)
finally:
volume.size = original_size
volume.save()
..........在
_create_from_image_cache_or_download 中会将镜像下载到本地临时文件，再通过 qemu 获取info信息，最终调用了
_create_from_image_download。在
_create_from_image_download中调用的本backend的driver执行create_volume 操作，并调用 copy_image_to_volume将镜像数据写入到volume
def _create_from_image_download(self, context, volume, image_location,
image_meta, image_service):
..........
model_update = self.driver.create_volume(volume) or {}
self._cleanup_cg_in_volume(volume)
model_update['status'] = 'downloading'
try:
volume.update(model_update)
volume.save()
except exception.CinderException:
LOG.exception("Failed updating volume %(volume_id)s with "
"%(updates)s",
{'volume_id': volume.id,
'updates': model_update})
try:
volume_utils.copy_image_to_volume(self.driver, context, volume,
image_meta, image_location,
image_service)
except exception.ImageTooBig:
with excutils.save_and_reraise_exception():
LOG.exception("Failed to copy image to volume "
"%(volume_id)s due to insufficient space",
{'volume_id': volume.id})
return model_update
def copy_image_to_volume(driver, context, volume, image_meta, image_location,
image_service):
..........
try:
image_encryption_key = image_meta.get('cinder_encryption_key_id')
if volume.encryption_key_id and image_encryption_key:
# If the image provided an encryption key, we have
# already cloned it to the volume's key in
# _get_encryption_key_id, so we can do a direct copy.
driver.copy_image_to_volume(
context, volume, image_service, image_id)
elif volume.encryption_key_id:
# Creating an encrypted volume from a normal, unencrypted,
# image.
driver.copy_image_to_encrypted_volume(
context, volume, image_service, image_id)
else:
driver.copy_image_to_volume(
context, volume, image_service, image_id)这里调用了腾凌存储的
cinder.volume.drivers.tengling.tengling_driver.TenglingISCSIDriver.create_volume 创建云盘，并调用腾凌driver执行了copy_image_to_volume。这里TenglingISCSIDriver继承自了driver.ISCSIDriver ，所以调用了原生ISCSIDriver的
def create_volume(self, volume):
"""Create a volume."""
volume_type = self._get_volume_type(volume)
opts = self._get_volume_params(volume_type)
if (opts.get('hypermetro') == 'true'
and opts.get('replication_enabled') == 'true'):
err_msg = _("Hypermetro and Replication can not be "
"used in the same volume_type.")
LOG.error(err_msg)
raise exception.VolumeBackendAPIException(data=err_msg)
lun_params, lun_info, model_update = (
self._create_base_type_volume(opts, volume, volume_type))
model_update = self._add_extend_type_to_volume(opts, lun_params,
lun_info, model_update)
return model_update原生ISCSIDriver 会通过os_brick模块远程iscsi挂载磁盘到本地
class ISCSIDriver(VolumeDriver):
.........
def copy_image_to_volume(self, context, volume, image_service, image_id):
"""Fetch image from image_service and write to unencrypted volume.
This does not attach an encryptor layer when connecting to the volume.
"""
self._copy_image_data_to_volume(
context, volume, image_service, image_id, encrypted=False)
.........
def _copy_image_data_to_volume(self, context, volume, image_service,
image_id, encrypted=False):
"""Fetch the image from image_service and write it to the volume."""
LOG.debug('copy_image_to_volume %s.', volume['name'])
use_multipath = self.configuration.use_multipath_for_image_xfer
enforce_multipath = self.configuration.enforce_multipath_for_image_xfer
properties = utils.brick_get_connector_properties(use_multipath,
enforce_multipath)
attach_info, volume = self._attach_volume(context, volume, properties) # 这里会挂载远程disk
try:
if encrypted:
encryption = self.db.volume_encryption_metadata_get(context,
volume.id)
utils.brick_attach_volume_encryptor(context,
attach_info,
encryption)
try:
image_utils.fetch_to_raw(
context,
image_service,
image_id,
attach_info['device']['path'],
self.configuration.volume_dd_blocksize,
size=volume['size']) #这里写入镜像数据
except exception.ImageTooBig:
with excutils.save_and_reraise_exception():
LOG.exception("Copying image %(image_id)s "
"to volume failed due to "
"insufficient available space.",
{'image_id': image_id})
finally:
if encrypted:
utils.brick_detach_volume_encryptor(attach_info,
encryption)
finally:
self._detach_volume(context, attach_info, volume, properties,
force=True)fetch_to_volume_format 中会fetch镜像文件到本地临时目录，最后通过执行qemu-img convert 命令将镜像临时文件数据写入到本地iscsi远程磁盘中，至此虚机的系统盘数据写入完成。
def fetch_to_volume_format(context, image_service,
image_id, dest, volume_format, blocksize,
volume_subformat=None, user_id=None,
project_id=None, size=None, run_as_root=True):
.........
convert_image(tmp, dest, volume_format,
out_subformat=volume_subformat,
src_format=disk_format,
run_as_root=run_as_root)
def _convert_image(prefix, source, dest, out_format,
out_subformat=None, src_format=None,
run_as_root=True, cipher_spec=None, passphrase_file=None):
cmd = _get_qemu_convert_cmd(source, dest,
out_format=out_format,
src_format=src_format,
out_subformat=out_subformat,
cache_mode=cache_mode,
prefix=prefix,
cipher_spec=cipher_spec,
passphrase_file=passphrase_file) #拼接 qemu-img convert 命令最后cinder侧更新数据库，至此cinder侧工作完成了，概括性的流程如下:
2.2 nova侧实现
nova侧在给虚机挂载磁盘时，nova-compute收到请求后attach volume的请求后调用nova.compute.manager.ComputeManager.attach_volume
def _attach_volume(self, context, instance, bdm):
context = context.elevated()
LOG.info('Attaching volume %(volume_id)s to %(mountpoint)s',
{'volume_id': bdm.volume_id,
'mountpoint': bdm['mount_device']},
instance=instance)
compute_utils.notify_about_volume_attach_detach(
context, instance, self.host,
action=fields.NotificationAction.VOLUME_ATTACH,
phase=fields.NotificationPhase.START,
volume_id=bdm.volume_id)
try:
bdm.attach(context, instance, self.volume_api, self.driver,
do_driver_attach=True)
................nova.virt.block_device.
DriverVolumeBlockDevice.attach中会 调用 _legacy_volume_attach 进行挂载磁盘
@update_db
def attach(self, context, instance, volume_api, virt_driver,
do_driver_attach=False, **kwargs):
..................
# Check to see if we need to lock based on the shared_targets value.
# Default to False if the volume does not expose that value to maintain
# legacy behavior.
if volume.get('shared_targets', False):
# Lock the attach call using the provided service_uuid.
@utils.synchronized(volume['service_uuid'])
def _do_locked_attach(*args, **_kwargs):
self._do_attach(*args, **_kwargs)
_do_locked_attach(context, instance, volume, volume_api,
virt_driver, do_driver_attach)
else:
# We don't need to (or don't know if we need to) lock.
self._do_attach(context, instance, volume, volume_api,
virt_driver, do_driver_attach)
def _do_attach(self, context, instance, volume, volume_api, virt_driver,
do_driver_attach):
"""Private method that actually does the attach.
This is separate from the attach() method so the caller can optionally
lock this call.
"""
context = context.elevated()
connector = virt_driver.get_volume_connector(instance)
if not self['attachment_id']:
self._legacy_volume_attach(context, volume, connector, instance,
volume_api, virt_driver,
do_driver_attach)
else:
self._volume_attach(context, volume, connector, instance,
volume_api, virt_driver,
self['attachment_id'],
do_driver_attach)legacyvolume_attach 会 调用cinder 的 initialize_connection 获取volume挂载的connection info，这里因为是iscsi 云盘，cinder会返回iSCSI的 connection info。
def _legacy_volume_attach(self, context, volume, connector, instance,
volume_api, virt_driver,
do_driver_attach=False):
volume_id = volume['id']
connection_info = volume_api.initialize_connection(context,
volume_id,
connector)
..................
# If do_driver_attach is False, we will attach a volume to an instance
# at boot time. So actual attach is done by instance creation code.
if do_driver_attach:
encryption = encryptors.get_encryption_metadata(
context, volume_api, volume_id, connection_info)
try:
virt_driver.attach_volume(
context, connection_info, instance,
self['mount_device'], disk_bus=self['disk_bus'],
device_type=self['device_type'], encryption=encryption)
except Exception:
with excutils.save_and_reraise_exception():
LOG.exception("Driver failed to attach volume "
"%(volume_id)s at %(mountpoint)s",
{'volume_id': volume_id,
'mountpoint': self['mount_device']},
instance=instance)
volume_api.terminate_connection(context, volume_id,
connector)
..................
if volume['attach_status'] == "detached":
# NOTE(mriedem): save our current state so connection_info is in
# the database before the volume status goes to 'in-use' because
# after that we can detach and connection_info is required for
# detach.
self.save()
try:
volume_api.attach(context, volume_id, instance.uuid,
self['mount_device'], mode=mode)virt_driver.attach_volume 会调用libvirt 挂载磁盘 ，self._connect_volume 会依据connection info判断为iSCSI driver 会Calling os-brick to attach iSCSI Volume ，最终libvirt 在线添加了disk到虚机里面，至此虚机侧挂载完成！
def attach_volume(self, context, connection_info, instance, mountpoint,
disk_bus=None, device_type=None, encryption=None):
guest = self._host.get_guest(instance)
disk_dev = mountpoint.rpartition("/")[2]
bdm = {
'device_name': disk_dev,
'disk_bus': disk_bus,
'device_type': device_type}
..................
self._connect_volume(context, connection_info, instance,
encryption=encryption)
..................
try:
state = guest.get_power_state(self._host)
live = state in (power_state.RUNNING, power_state.PAUSED)
guest.attach_device(conf, persistent=True, live=live)总结:
IP SAN 由于基于传统IP网络，所以优势是成本低、易维护且比较灵活，但同时也限制了他只适合应用于对于性能要求并不是太高的场景，性能上比较不如FS SAN以及nvmf等，因此IP SAN 适合于中小型应用场景，所以需要针对不同的使用场景下进行选择不同的存储协议。
本文借以兼容腾凌存储梳理了openstack侧虚机挂载使用商业 IP SAN存储的整个流程，对于理解包括支持其他类型的SAN/NVMF等存储有一定的参考意义。
相关文章
- c#.Net.NetCore面试(四十四)c#/net/netcore读取文件
- iNeuOS工业互联网操作系统，从NetCore3.1升级到Net6的过程汇报
- c#.Net.NetCore面试(二十二)c# Hashtable(哈希表)解析
- 开发现代化的.NetCore控制台程序:(4)使用GithubAction自动构建以及发布nuget包
- Netcore部署和使用docker
- .NetCore中使用分布式事务DTM的二阶段消息
- 3-跟我一起学.NetCore之依赖注入
- 微软停止更新 .NET Standard 今后转向.NET 5
- c#.Net.NetCore面试(六)readonly与const区别?
- c#.Net.NetCore面试(三十九)选择排序算法