Bug #4924

ceph: tm migration scripts are killing vms with shared system datastore

Added by Laurent Grawet over 4 years ago. Updated about 4 years ago.

Status:ClosedStart date:11/16/2016
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Drivers - Storage
Target version:Release 5.4
Resolution:fixed Pull request:
Affected Versions:OpenNebula 5.2

Description

Hi,

There is a huge problem in remotes/tm/ceph/premigrate

ssh_make_path "$DST_HOST" "$DST_PATH" 

log "Moving $SRC_HOST:$DST_PATH to $DST_HOST:$DST_PATH" 

ssh_exec_and_log "$DST_HOST" "rm -rf '$DST_PATH'" \
    "Error removing target path to prevent overwrite errors" 

TAR_COPY="$SSH $SRC_HOST '$TAR -C $DST_PATH_DIRNAME --sparse -cf - $DST_PATH_BASENAME'" 
TAR_COPY="$TAR_COPY | $SSH $DST_HOST '$TAR -C $DST_PATH_DIRNAME --sparse -xf -'"

and also probably in remotes/tm/ceph/mv

sh_make_path "$DST_HOST" "$DST_DIR" "ssh" 

log "Moving $SRC to $DST" 

ssh_exec_and_log "$DST_HOST" "rm -rf '$DST_PATH'" \
    "Error removing target path to prevent overwrite errors" 

TAR_SSH=$(cat <<EOF
set -e -o pipefail

$TAR -C $SRC_DS_DIR --sparse -cf - $SRC_VM_DIR | $SSH $DST_HOST '$TAR -C $DST_DIR --sparse -xf -'
rm -rf $SRC_PATH
EOF
)

This may be fine while using "ssh" transfer mode on system datastore but when using "shared" mode, it is removing the vm directory in the system datastore, including volatile disks, leading to a VM crash and/or inability to migrate because of missing disk and deployment files.

Don't you think all these operations should be handled in remotes/tm/ssh instead of ceph driver ?

Thanks for your help,

Laurent

Associated revisions

Revision 01968320
Added by Anton Todorov over 4 years ago

tm/ceph {pre,post}migrate do nothing if not SYSTEM DS

The proposed solution partially addresses bug #4924

IMO in case of SYSTEM DS on shared filesystem the TM_MAD must be 'shared'.
In this case the tm/ceph/mv will not be called but then {pre,post}migrate
is called by migrate-other. In that case there is no work for Ceph.

History

#1 Updated by Laurent Grawet over 4 years ago

There is also a problem in remotes/tm/ceph/postmigrate

exec_and_log "$SSH $SRC_HOST rm -rf $DST_PATH"

#2 Updated by Anton Todorov over 4 years ago

IMHO you must use `shared` TM_MAD for SYSTEM datastore. This will address tm/ceph/mv as it will not be called.

There is still a bug in {pre,post}migrate for which I am proposing a solution in pull-request https://github.com/OpenNebula/one/pull/160

In brief when the {pre,post}migrate are called in 'slave' context (the master is tm/shared/{pre,post}migrate) the scripts must do nothing.

The patch is adding just this check:

if [ -n "$7" ]; then
    log "Called from tm/$7 but I've nothing to do" 
    exit 0
fi

Kind Regards,
Anton Todorov

#3 Updated by Ruben S. Montero over 4 years ago

  • Target version set to Release 5.4

#4 Updated by Jaime Melis over 4 years ago

  • Status changed from Pending to Closed
  • Resolution set to invalid

Hi Laurent, as Anton has explained, you are probably using TM_MAD=ceph, but you need to use TM_MAD=shared if you have a shared file system. For TM_MAD=ceph there can't be a shared file system.

It's true that the documentation doesn't explain it very clearly, it's in the Note banner in this section:
http://docs.opennebula.org/5.2/deployment/open_cloud_storage_setup/ceph_ds.html#create-a-system-datastore

I will change the docs to reflect this more clearly.

Closing as invalid (if you have any troubles with this problem using TM_MAD=shared please reopen).

#5 Updated by Laurent Grawet over 4 years ago

Hi Jaime,

Thanks for clarifying this! (TM_MAD must be set to "shared" in my case)
I haven't read this part of the 5.2 doc as I'm coming from 4.14.2.
It was not the case in this version (ceph was also not supported as system ds)
http://docs.opennebula.org/4.14/administration/storage/ceph_ds.html

Configuring Ceph Datastores
...

TM_MAD    Transfer drivers for the datastore, use ceph, see below

It looks to me it has changed with 5.x so this should also be added to release notes.

And what about Anton's patch to avoid a recursive call from other drivers ?

Thank you both,

Laurent

#6 Updated by Jaime Melis about 4 years ago

  • Status changed from Closed to New

Reopening, there might be a problem indeed.

#7 Updated by Anton Todorov about 4 years ago

Our storage addon is supporting both ssh and shared modes switchable via custom datastore attribute (SP_SYSTEM in our case).
For historical reasons the default is shared mode and there is no cleanup step but when there is SP_SYSTEM=ssh we do cleanup.

Best Regards

#8 Updated by Vlastimil Holer about 4 years ago

  • Status changed from New to Closed

Anton Todorov wrote:

IMHO you must use `shared` TM_MAD for SYSTEM datastore. This will address tm/ceph/mv as it will not be called.

There is still a bug in {pre,post}migrate for which I am proposing a solution in pull-request https://github.com/OpenNebula/one/pull/160

In brief when the {pre,post}migrate are called in 'slave' context (the master is tm/shared/{pre,post}migrate) the scripts must do nothing.

The patch is adding just this check:

Thank you for the PR #160, it was merged by Jaime today.

So, if Ceph is used with the shared filesystem and TM_MAD=shared for the system DS, there shouldn't be problem anymore.

I'm closing the bug. Thank you.

#9 Updated by Vlastimil Holer about 4 years ago

  • Resolution changed from invalid to fixed

Also available in: Atom PDF