Can You Do More Than 255?
8 February 2018
In my role here at Datadobi, talking about moving, replicating, and syncing unstructured data all the time, I cannot begin to tell you how often I hear people ask if we can handle long file names or long path names. They of course ask the question for a valid reason. Many legacy tools cannot handle paths over 255 characters.
The short answer?
Our DobiMiner Suite doesn’t have a limit as long as the source system, target system, or the protocol in-use doesn’t have a limit.
The longer answer: with step-by-step-proof
Step 1: Create some sample data
Create a 2G filesystem on a VNX:
Mount the filesystem:
Create an export of it:
Now write a simple shell script/loop to create a dataset: (60 directories wide by 60 directories deep). The file name for the curious is the first line of the first page from War and Peace by Leo Tolstoy.
Run the script:
Files successfully created:
Step 2: In DobiMigrate, setup a migration of this filesystem
Create a Migration from (in this case, the VNX, to Isilon):
Perform the migration (with NFS in this case, but SMB works the same for this test):
Step 3: Evaluate the results
Look at the structure and data integrity
So here is a file that is 61+ levels deep in the source VNX structure:
Here is the MD5 hash of that file on the source:
Here is the preserved structure on the target:
Here is the same file on the target:
Notice the MD5 hashes match exactly, as do the paths relative to what we picked as the path on the source and path on the target.
Look at the Character Count
Frequently in migrations many legacy tools run up against a character limit of 255 characters for the entire path.
So, let’s just take a quick view on how many characters are actually in the path we migrated above (excluding the hostname). From the root of the VNX control station to the root of the filesystem there are 29 characters.
I picked a single path to go down and here is the full structure:
So, that’s 600 characters of folder names. 629-29 (how the control station sees it) = 600. Plus another 86 characters for the filename within that folder.
In Summary
We successfully migrated a file that was 60 levels deep with a file with 86 characters in the filename. That’s a total of 686 characters in the path, not including the server name or domain name, and we handled it without a problem. As you can see above the file checksum is also identical source-to-target. So to the question of our title above: “Can you do more than 255?” with DobiMigrate (and our whole DobiMiner suite) the answer is unequivocally YES!
Appendix: Limits on Different NAS Systems & Protocols
Dell EMC VNX
VNX has a limit of 255 bytes over NFS but 255 characters over SMB/CIFS. (See page 9 of this guide for more information): https://support.emc.com/docu32528_NAS-Support-Matrix.pdf
Dell EMC Isilon
Isilon’s OneFS has a limit (currently) of 255 bytes in a filename. (See page 13 of this guide for more information): https://www.emc.com/collateral/TechnicalDocument/docu84267.pdf
NetApp ONTAP9
ONTAP has a limit of 255 characters in a filename regardless of protocol:
https://kb.netapp.com/app/answers/answer_view/a_id/1000976
Another helpful page on NetApp ONTAP 9 that explains a number of these concepts very well:
https://library.netapp.com/ecm/ecm_get_file/ECMLP2494081
Protocol Limitations
Ultimately these limits are all bound by both the storage system and the protocol.
SMB
http://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx#maxpath
“A pathname MUST be no more than 32,760 characters in length.”
“Each pathname component MUST be no more than 255 characters in length.”
https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file
Talks about all the different standards and reasons for them such as historical 8.3 format, etc.
NFS
https://tools.ietf.org/html/rfc1813
“Any POSIX limitations in the remote server filesystem will be passed through to the client. For example, the length of filenames may vary across servers from different operating systems. NFS (versions 2 and 3) limits filenames to 255 characters; mountd (versions 1 and 3) limits pathnames to 1024 characters.”