Trouble-shooting Methodologies and Errors in DobiMigrate

Scenario

You have DobiMigrate installed, had some firewall ports opened to discover that last fileserver, straightened out security for your SMB credentials, and decided on the path structure on new hardware, and it’s clear sailing from here.  Your migrations are all set up, files are copying, and you’re feeling confident about your data migration endeavor. Nevertheless, now is not the time to rest—the success of your migration depends on preparation and investigating the details so there are no surprises on Sunday morning at 3am.

The first thing is to check the obvious. Is DobiMigrate reporting any errors? If so, what are they?  Sure, most migrations will have errors—reporting files that are in use and copy errors due to temporary files being deleted between the time of the filesystem scan and copy—but sometimes looking a little deeper into the errors resolves issues before they become problems on switchover.  Something that should always stand out is a consistent number of errors across the iterations, suggesting that these are probably not transient errors and will need to be dealt with.  Take some time and look at the error details and see what category they fit into, then investigate further by clicking the exclamation point to bring up the details of each error.

A Primer on DobiMigrate Error Conditions

Permission-denied Errors

These can be difficult to isolate because they can occur on source or target and can be on reads or writes.  First, identify the fileserver and type of error.  Permission errors on reads could be that the SMB credentials need to be added to the local Administrators and local Backup Operators groups on the fileserver, or the lack of share permissions for the SMB user credentials if you’re using existing shares, rather than the DobiMigrate auto-created hidden migration shares (for example, DM_ebb3e728-0ab9-48b9-b248-bdf3f15d34c8-Xr-8XM8MRONlP5qgIg6qWQ$).  NFS permission errors are often due to exports with root squash enabled, so double check on the source and target fileserver.

Out of Space Errors

These are usually easy to solve.  They can occur if quotas are enabled and misconfigured for the dataset size being migrated or the destination fileserver runs out of space due to deduplication, compression, or archiving lag.

File Not Found Errors

Usually this is due to a file being present upon a scan but deleted by the time the copy task is requested.  If this is the case, these errors will come and go and change between iterations.  If the error amount is consistent, there can be other things that show up as ‘file not found’, one being character set encoding mismatches between fileservers (ie: utf-8 vs iso-8859-1).  Luckily DobiMigrate does encoding pre-checks when configuring a new migration to help ensure that these don’t crop up in the first place.

File in Use Errors

It is almost always just that—files that are opened by other users.  These are usually not an area for concern because they will copy during the switchover when the shares and exports go read only and the user connections are terminated.

Timeouts and Network Errors

Network errors can mean a lot of things and can be tricky, so you need to investigate them carefully.  They can be due to an intermittent or overloaded network connection between the proxy and a file server or communication issues with a domain controller.  Use traditional network tools to help troubleshoot these errors, and when things get difficult, a Wireshark capture may be a good tactic to discover what is happening to the packets.

Other

The rest of the errors get lumped into the ‘Other’ category, and their nature can range widely.  You can find things like file names too long for the destination fileserver, broken pipes unable to be copied, or files that are just plain invalid for some other reason.

Now is Not the Time to Panic

Migrations can be a bit like spring cleaning, you will find odd things tucked away in corners that need to be remediated. Just stick to basic troubleshooting techniques to discover what the issue really is and how to fix it.

Start by creating the shares and exports ahead of time in DobiMigrate under the Settings tile of the Migrations tab. They need to be created before the switchover, so this is a good time to create them read-only for testing, and gives you a chance to deal with any issues that may surface.  DobiMigrate shows the status of the source and destination shares and exports along with any compatibility issues.  Shown below is one type of compatibility issue that may occur when migrating between different fileserver platforms.

Once the exports and shares are created read-only, open the export or share with native tools like Windows File Explorer or a Unix shell and look at the file in question, examine its permissions, see if you can copy it somewhere, see if you can open the file, as invalid files will usually exhibit the same behavior outside of the migration.  Always use the same user credentials as the migration when testing if possible, as being able to access a file as the owner or some other credentials does not guarantee the SMB credentials entered into DobiMigrate will have access.  Here is an example of checking the permissions of a file that has been migrated from a VNX fileserver to an Isilon within Windows File Explorer:

[root@dmnfsproxy data]# pwd
/mnt/UATIsilon/data[root@dmnfsproxy Folderwindows]# ls -la
total 79
drwxrwxrwx 2 10009 10000 246 Mar 16 01:11 .
drwxrwxrwx 5 root  root  336 Mar 16 01:32 ..
-rwx------ 1 10009 10000  30 Mar 16 00:54 Createdbywindows_everyone_Readonly.txt
-r-x------ 1 10009 10000  30 Mar 16 00:54 Createdbywindows_everyone.txt
-rwxrwxrwx 1 10009 10000  30 Mar 16 00:54 Createdbywindows_owneronly_Readonly.txt
-rwxrwxrwx 1 10003 10000   6 Mar 13 21:05 Createdbywindows_owneronly.txt
-rwxrwxr-x 1 10009 10000  43 Mar 13 19:44 Createdbywindows.txt