In the last article, titled How to Recursively Copy a Folder (Directory) in Python, I covered how to copy a folder recursively from one place to another.
As useful as that is, there is still something else that we could add that would make copying a directory much more user friendly!
Enter the progress bar from stage left. It's a useful utility that all big name software uses to tell the user something is happening and where it is with the task it is doing.
So how do we incorporate this into copying a directory? Glad you asked, my question filled friend!
The Progress Bar
First, we'll make a simple class to make printing progress 100 times easier as well as give us a nice reusable class to use in other projects that might need it.
Here we go. First we'll add the __init__ method:
[python]
# -*- coding: utf-8 -*-
import sys
class ProgressBar(object):
def __init__(self, message, width=20, progressSymbol=u'▣ ', emptySymbol=u'□ '):
self.width = width
if self.width < 0:
self.width = 0
self.message = message
self.progressSymbol = progressSymbol
self.emptySymbol = emptySymbol
[/python]
That wasn't too hard! We pass in the message we want to print, the width of the progress bar, and the two symbols for progress done and empty progress. We check that the width is always greater that or equal to zero, because we can't have a negative length! 🙂
Simple stuff.
Alright, on to the hard part. We have to figure out how to update and render the progress bar to the output window without printing a new line every single time.
That's where the carriage return (\r) comes in. The carriage return is a special character that allows the printed message to start at the beginning of a line when printed. Every time we print something, we'll just use it at the beginning of our line and it should take care of our needs just fine. The only limitation this has is that if the progress bar happens to wrap in the terminal, the carriage return will not work as expected and will just output a new line.
Here is our update function that will print the updated progress:
[python]
def update(self, progress):
totalBlocks = self.width
filledBlocks = int(round(progress / (100 / float(totalBlocks)) ))
emptyBlocks = totalBlocks - filledBlocks
progressBar = self.progressSymbol * filledBlocks + \
self.emptySymbol * emptyBlocks
if not self.message:
self.message = u''
progressMessage = u'\r{0} {1} {2}%'.format(self.message,
progressBar,
progress)
sys.stdout.write(progressMessage)
sys.stdout.flush()
def calculateAndUpdate(self, done, total):
progress = int(round( (done / float(total)) * 100) )
self.update(progress)
[/python]
What we just did is calculate the number of filled blocks using the total blocks needed. It looks a little complicated, so let me break it down. We want the number of filled blocks to be the progress divided by 100% divided by the total number of blocks. The calculateAndUpdate function is just to make our life a little easier. All it does is calculate and print the percentage done given the current number of items and the total number of items.
[python]
filledBlocks = int(round(progress / (100 / float(totalBlocks)) ))
[/python]
That call to float is there to make sure the calculation is done with floating point precision, and then we round up the floating point number with round and make it an integer with int.
[python]
emptyBlocks = totalBlocks - filledBlocks
[/python]
Then the empty blocks are simply the totalBlocks minus the filledBlocks.
[python]
progressBar = self.progressSymbol * filledBlocks + \
self.emptySymbol * emptyBlocks
[/python]
We take the number of filled blocks and multiply it by the progressSymbol and add it to the emptySymbol multiplied by the number of empty blocks.
[python]
progressMessage = u'\r{0} {1} {2}%'.format(self.message,
progressBar,
progress)
[/python]
We use the carriage return at the beginning of the message to set the print position to the beginning of the line and make a message formatted like: "Message! ▣ ▣ ▣ □ □ □ 50%".
[python]
sys.stdout.write(progressMessage)
sys.stdout.flush()
[/python]
And, finally, we print the message to the screen. Why don't we just use the print function? Because the print function adds a new line to the end of our message and that's not what we want. We want it printed just as we set it.
On with the show! To the copying a directory function!
Copying a Directory (Folder) with Progress in Python
In order to know our progress, we'll have to sacrifice some speed. We need to count all of the files that we're copying to get the total amount of progress left, and this requires us to recursively search the directory and count all of the files.
Let's write a function to do that.
[python]
import os
def countFiles(directory):
files = []
if os.path.isdir(directory):
for path, dirs, filenames in os.walk(directory):
files.extend(filenames)
return len(files)
[/python]
Pretty straight forward. We just checked if the directory passed in is a directory, then if it is, recursively walk the directories and add the files in the directories to the files list. Then we just return the length of the list. Simple.
Next is the copying directories function. If you looked at the article mentioned at the top of this article, you'll notice that I used the shutil.copytree function. Here we can't do that because shutil doesn't support progress updates, so we'll have to write our own copying function.
Let's do it!
First we create an instance of our ProgressBar class.
[python]
p = ProgressBar('Copying files...')
[/python]
Next, we define a function to make directories that don't exist yet. This will allow us to create the directory structure of the source directory.
[python]
def makedirs(dest):
if not os.path.exists(dest):
os.makedirs(dest)
[/python]
If the directory doesn't exist, we create it.
Now for the copy function. This one is a bit of a doozy, so I'll put it all down in one place, then break it down.
[python]
import shutil
def copyFilesWithProgress(src, dest):
numFiles = countFiles(src)
if numFiles > 0:
makedirs(dest)
numCopied = 0
for path, dirs, filenames in os.walk(src):
for directory in dirs:
destDir = path.replace(src,dest)
makedirs(os.path.join(destDir, directory))
for sfile in filenames:
srcFile = os.path.join(path, sfile)
destFile = os.path.join(path.replace(src, dest), sfile)
shutil.copy(srcFile, destFile)
numCopied += 1
p.calculateAndUpdate(numCopied, numFiles)
print
[/python]
Firstly, we count the number of files.
[python]
def copyFilesWithProgress(src, dest):
numFiles = countFiles(src)
[/python]
Nothing too difficult about that.
Next, if there are actually files to copy, we create the destination folder if it doesn't exist and initialize a variable to count the current number of files copied.
[python]
if numFiles > 0:
makedirs(dest)
numCopied = 0
[/python]
Then, we walk the directory tree and create all the directories needed.
[python]
for path, dirs, filenames in os.walk(src):
for directory in dirs:
destDir = path.replace(src, dest)
makedirs(os.path.join(destDir, directory))
[/python]
The only tricky part here is that I replaced the source directory string in path, which is the root path, with the destination folder path.
Next, we copy files and update progress!
[python]
for path, dirs, filenames in os.walk(src):
(...)
for sfile in filenames:
srcFile = os.path.join(path, sfile)
destFile = os.path.join(path.replace(src, dest), sfile)
shutil.copy(srcFile, destFile)
numCopied += 1
p.calculateAndUpdate(numCopied, numFiles)
[/python]
Here, we go through all the files, copy them, update the number of files copied so far, and then draw our progress bar.
That's it! We're done! Now you have a nice copying function that will alert you of the progress while copying files.
Moving a Directory (Folder) with Progress in Python
As a note, you may also change the shutil.copy function call in copyFilesWithProgress to shutil.move and have a move progress bar instead. All the other code would be the same, so I'm not going to rewrite it here. I'll leave that up to you! 🙂
Until the next time, I bid you adieu.
