DevOps — Check Integrity of Old School FTP/SFTP Folder Transfer with 2 Commands

Use case

You want to transfer the build folder that normally have 100–1,000 files and

Transfer tool ranging from the old school WinSCP, FileZilla, SCP and SFTP command.

Problem

Sometimes you discover that files are not all transferred

So I will put two commands below on the deploy script when build and when deploy to check the integrity of the transfer.

Solution

1. Count the file

Work on both Linux, Mac and WSL

find ./out -type f | wc -l

2. MD5 Sum of all File MD5 Hash

We calculate the MD5 of all files

Then sort it so the output will always be the same

Finally MD5 sum of all sorted item

find ./out -type f -exec md5sum {} \; | sort | md5sum

On Mac BSD find provide the sort option as -s and md5 command with -q option to only output the hash

find -s ./out -type f -exec md5 -q {} \; | md5 -q

3. The Tar Ball

Most program is not very good at queing the file

You need to pack it like on ball

# Pack
tar -zcf out.tar.gz ./out 

# Unpack
tar -zxf out.tar.gz

Bonus

Unpacking to the old location will override the file from old pack

but NOT REMOVE the file that exist on the old one and not on the new one.

Always do the clean before unpack

rm -rf ./out