Trouble-shooting Methodologies and Errors in DobiMigrate
4 April 2017
Scenario
You have DobiMigrate installed, had some firewall ports opened to discover that last fileserver, straightened out security for your SMB credentials, and decided on the path structure on new hardware, and it’s clear sailing from here. Your migrations are all set up, files are copying, and you’re feeling confident about your data migration endeavor. Nevertheless, now is not the time to rest—the success of your migration depends on preparation and investigating the details so there are no surprises on Sunday morning at 3am.
The first thing is to check the obvious. Is DobiMigrate reporting any errors? If so, what are they? Sure, most migrations will have errors—reporting files that are in use and copy errors due to temporary files being deleted between the time of the filesystem scan and copy—but sometimes looking a little deeper into the errors resolves issues before they become problems on switchover. Something that should always stand out is a consistent number of errors across the iterations, suggesting that these are probably not transient errors and will need to be dealt with. Take some time and look at the error details and see what category they fit into, then investigate further by clicking the exclamation point to bring up the details of each error.
A Primer on DobiMigrate Error Conditions
Permission-denied Errors
These can be difficult to isolate because they can occur on source or target and can be on reads or writes. First, identify the fileserver and type of error. Permission errors on reads could be that the SMB credentials need to be added to the local Administrators and local Backup Operators groups on the fileserver, or the lack of share permissions for the SMB user credentials if you’re using existing shares, rather than the DobiMigrate auto-created hidden migration shares (for example, DM_ebb3e728-0ab9-48b9-b248-bdf3f15d34c8-Xr-8XM8MRONlP5qgIg6qWQ$). NFS permission errors are often due to exports with root squash enabled, so double check on the source and target fileserver.
Out of Space Errors
These are usually easy to solve. They can occur if quotas are enabled and misconfigured for the dataset size being migrated or the destination fileserver runs out of space due to deduplication, compression, or archiving lag.
File Not Found Errors
Usually this is due to a file being present upon a scan but deleted by the time the copy task is requested. If this is the case, these errors will come and go and change between iterations. If the error amount is consistent, there can be other things that show up as ‘file not found’, one being character set encoding mismatches between fileservers (ie: utf-8 vs iso-8859-1). Luckily DobiMigrate does encoding pre-checks when configuring a new migration to help ensure that these don’t crop up in the first place.
File in Use Errors
It is almost always just that—files that are opened by other users. These are usually not an area for concern because they will copy during the switchover when the shares and exports go read only and the user connections are terminated.
Timeouts and Network Errors
Network errors can mean a lot of things and can be tricky, so you need to investigate them carefully. They can be due to an intermittent or overloaded network connection between the proxy and a file server or communication issues with a domain controller. Use traditional network tools to help troubleshoot these errors, and when things get difficult, a Wireshark capture may be a good tactic to discover what is happening to the packets.
Other
The rest of the errors get lumped into the ‘Other’ category, and their nature can range widely. You can find things like file names too long for the destination fileserver, broken pipes unable to be copied, or files that are just plain invalid for some other reason.
Now is Not the Time to Panic
Migrations can be a bit like spring cleaning, you will find odd things tucked away in corners that need to be remediated. Just stick to basic troubleshooting techniques to discover what the issue really is and how to fix it.
Start by creating the shares and exports ahead of time in DobiMigrate under the Settings tile of the Migrations tab. They need to be created before the switchover, so this is a good time to create them read-only for testing, and gives you a chance to deal with any issues that may surface. DobiMigrate shows the status of the source and destination shares and exports along with any compatibility issues. Shown below is one type of compatibility issue that may occur when migrating between different fileserver platforms.
Once the exports and shares are created read-only, open the export or share with native tools like Windows File Explorer or a Unix shell and look at the file in question, examine its permissions, see if you can copy it somewhere, see if you can open the file, as invalid files will usually exhibit the same behavior outside of the migration. Always use the same user credentials as the migration when testing if possible, as being able to access a file as the owner or some other credentials does not guarantee the SMB credentials entered into DobiMigrate will have access. Here is an example of checking the permissions of a file that has been migrated from a VNX fileserver to an Isilon within Windows File Explorer:
[root@dmnfsproxy UATCelerra]# pwd /mnt/UATCelerra [root@dmnfsproxy Folderwindows]# ls -la total 56 drwxr-xr-x 2 10009 10000 1024 Mar 16 01:11 . drwxr-xr-x 6 root root 1024 Mar 16 01:33 .. -rwx------ 1 10009 10000 30 Mar 16 00:54 Createdbywindows_everyone_Readonly.txt -r-x------ 1 10009 10000 30 Mar 16 00:54 Createdbywindows_everyone.txt -rwxrwxrwx 1 10009 10000 30 Mar 16 00:54 Createdbywindows_owneronly_Readonly.txt -------rwx 1 10003 10000 6 Mar 13 21:05 Createdbywindows_owneronly.txt -rwxr-xr-x 1 10009 10000 43 Mar 13 19:44 Createdbywindows.txt [root@dmnfsproxy data]# pwd /mnt/UATIsilon/data [root@dmnfsproxy Folderwindows]# ls -la total 79 drwxrwxrwx 2 10009 10000 246 Mar 16 01:11 . drwxrwxrwx 5 root root 336 Mar 16 01:32 .. -rwx------ 1 10009 10000 30 Mar 16 00:54 Createdbywindows_everyone_Readonly.txt -r-x------ 1 10009 10000 30 Mar 16 00:54 Createdbywindows_everyone.txt -rwxrwxrwx 1 10009 10000 30 Mar 16 00:54 Createdbywindows_owneronly_Readonly.txt -------rwx 1 10003 10000 6 Mar 13 21:05 Createdbywindows_owneronly.txt -rwxr-xr-x 1 10009 10000 43 Mar 13 19:44 Createdbywindows.txt
A few spot checks on some files and directories should be sufficient for SMB or NFS copies, but Multi-Protocol can be a bit more complex. Below is the same file copied with DobiMigrate SMB only, the other copied with NFS then SMB Multi Pass functionality, and as you can see the file not only has more and different security entries but also different permissions. The same with the POSIX permissions, see below:
[root@dmnfsproxy data]# pwd /mnt/UATIsilon/data[root@dmnfsproxy Folderwindows]# ls -la total 79 drwxrwxrwx 2 10009 10000 246 Mar 16 01:11 . drwxrwxrwx 5 root root 336 Mar 16 01:32 .. -rwx------ 1 10009 10000 30 Mar 16 00:54 Createdbywindows_everyone_Readonly.txt -r-x------ 1 10009 10000 30 Mar 16 00:54 Createdbywindows_everyone.txt -rwxrwxrwx 1 10009 10000 30 Mar 16 00:54 Createdbywindows_owneronly_Readonly.txt -rwxrwxrwx 1 10003 10000 6 Mar 13 21:05 Createdbywindows_owneronly.txt -rwxrwxr-x 1 10009 10000 43 Mar 13 19:44 Createdbywindows.txt
This scenario was designed to illustrate that simply copying each protocol’s permissions in a multi-protocol environment will not always generate the same permissions on the destination. The source VNX handles each permission type independently, so in this case the Windows permissions were reduced to full control by the file owner only, but the POSIX permissions were set to 007, giving read/write/execute access by everyone, but owner and group are granted no permissions. In this case, the SMB and POSIX permissions are basically opposing each other, but they are perfectly valid configuration on the VNX. However, when the file is copied to the Isilon, it is subjected to Isilon’s Unified Security Model, which merges permissions and maps users into a single access token. The result in this case is an expanded ACL, and different POSIX permission than on the source VNX.
Migrating multi-protocol data between different fileservers with different ways of handling it may mean making some compromises in permissions to deliver similar access to the user community on the new platform. What is the right choice? Start with defining how the data is written and accessed before deciding on what protocol to approach the migration. For instance, if a filesystem is used mainly by CIFS users, but is also accessed by an export to a few Unix desktops to read a few text files, it would be wise to start with an SMB only copy, ensure the users are mapped correctly, and see if they have file access on the destination fileserver based on the ACLs. In many instances, the files are stored as single security style with the other protocol’s permissions being generated “on the fly” and are not good candidates for a DobiMigrate multi-pass migration.
In short, use the error reporting within DobiMigrate to identify and catalog the errors, and then do the investigation required to resolve each of them. Create the read only shares and exports on the destination fileservers ahead of time and invite a select few users to do some testing ahead prior to the switchover. In the end, it is diligent testing of a small dataset using the techniques discussed here that will give you confidence that the switchover event will go as smooth and as fast as possible, and the users will be able to access their files on Monday morning.