Page 1 of 3

Upload Vesta user backups to Dropbox automatically daily

Posted: Fri Mar 17, 2017 2:49 pm
by vesta_mtl
I have setup my Centos 7 server to send the Vesta backups of all my websites (users) to Dropbox automatically every day (after Vesta has run the backup job). In case it helps anyone, here is how.

Configure Dropbox uploader
  1. Go here: https://www.dropbox.com/developers/apps
  2. Create a new Dropbox app and give it access to a new folder in the Apps directory (e.g. server_backups)
  3. Generate and copy your access token
  4. On your server, enter these commands (make a directory, navigate to it, download the dropbox upload bash script dropbox_uploader.sh, give it execution permissions, execute it) and paste your token when prompted to.

Code: Select all

cd /
mkdir dropbox
cd dropbox
curl "https://raw.githubusercontent.com/andreafabrizi/Dropbox-Uploader/master/dropbox_uploader.sh" -o dropbox_uploader.sh
chmod 755 dropbox_uploader.sh
./dropbox_uploader.sh
Optional: Send a single file to Dropbox as a test (replace with your backup filename).

Code: Select all

/dropbox/dropbox_uploader.sh upload "/home/backup/admin.2017-03-17.tar" /

Schedule the backups to go to Dropbox daily

Make this new file:

Code: Select all

vi /usr/local/bin/send_site_backup_files_to_dropbox.sh
Add this to the file, then save it (type Esc, :wq):

Code: Select all

#!/bin/bash
#Save current date  as YYYY-MM-DD to a variable
DATE=$(date +"%Y-%m-%d")
#Loop through each file in the backup folder whose name has the current date
for X in /home/backup/*$DATE*; do
    #X is the filename with path. Remove path to get just the filename.
    NAME_NO_PATH=${X##*/}
    #Remove the date from the name (removes all text between the periods)
    NEW_NAME="${NAME_NO_PATH%%.*}.${NAME_NO_PATH##*.}"
    #Copy the file to tmp with the new non-dated name
    cp $X /tmp/$NEW_NAME
    #Send it to Dropbox
    /dropbox/dropbox_uploader.sh -f /root/.dropbox_uploader upload "/tmp/$NEW_NAME" /
    #Delete the file from tmp
    rm -rf /tmp/$NEW_NAME
done
Note: The backup file that is uploaded to Dropbox has the date removed from its filename so that every day, the new backup will overwrite the previous backup in Dropbox. This way you don't get an accumulation of backups (e.g. admin.2017-03-17.tar, admin.2017-03-18.tar, etc...) and always only have one backup file for each user (e.g. admin.tar) in Dropbox. Using Dropbox's file version history, you can access older backups of that user. The backups on the server are unchanged (they are not renamed, and they are kept for two days as per Vesta's normal behaviour).


Optional: Run it once as a test (your backups folder must have today's backups in it).

Code: Select all

sh /usr/local/bin/send_site_backup_files_to_dropbox.sh
Verify what time Vesta does its daily backups:
  1. Log into Vesta as admin
  2. Click Cron
  3. Find “sudo /usr/local/vesta/bin/v-backup-users” and click edit
  4. Verify what time Vesta does backups (change the time if desired)
Install moreutils (needed for "ts" command used in crontab):

Code: Select all

yum install moreutils
Edit crontab:

Code: Select all

crontab -e
Add this line to crontab so your backups are sent every morning at 3:45am (or any other time you want, just make sure it is at least 30 minutes after Vesta does the backups). Change the email address at the end to whatever address you wish to get the emailed report.

Code: Select all

45 03 * * * sh /usr/local/bin/send_site_backup_files_to_dropbox.sh | ts "[\%Y-\%m-\%d \%H:\%M:\%S]" 2>&1 | tee /var/log/send_backups_to_dropbox.log | mailx -s "Report for Vesta backup files sent to Dropbox" [email protected]
Optional: You can view the log here (but this will also get emailed to as per the previous step).

Code: Select all

cat /var/log/send_backups_to_dropbox.log
That is all, your backups will now be sent to Dropbox daily at the time you specified in crontab, and you will get an email report with the results.

I have successfully used this for files up to 10 Gb in size, and I think it should work fine for even larger. You can read more about how the Dropbox Uploader script does this, and other options it offers, here: https://github.com/andreafabrizi/Dropbox-Uploader

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Mon Mar 20, 2017 6:55 am
by skurudo
Nice manual, vesta_mtl
Sticky post now.

Did you test this solution on big files like 1-2-3-4-5-10Gb or more?

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Mon Mar 20, 2017 12:34 pm
by vesta_mtl
Thanks skurudo. Yes, I have uploaded files up to 10 Gb successfully. I have updated my post with this detail, and a link to the Dropbox Uploader script which explains how this is done, along with other options.

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Sun Apr 09, 2017 6:16 pm
by vikhyat
Thanks a lot for this.

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Tue May 02, 2017 1:35 pm
by pandabb
thank you will try this right away.

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Tue Oct 10, 2017 1:28 pm
by lpcs007
vesta_mtl wrote:I have setup my Centos 7 server to send the Vesta backups of all my websites (users) to Dropbox automatically every day (after Vesta has run the backup job). In case it helps anyone, here is how.

Configure Dropbox uploader
  1. Go here: https://www.dropbox.com/developers/apps
  2. Create a new Dropbox app and give it access to a new folder in the Apps directory (e.g. server_backups)
  3. Generate and copy your access token
  4. On your server, enter these commands (make a directory, navigate to it, download the dropbox upload bash script dropbox_uploader.sh, give it execution permissions, execute it).

Code: Select all

cd /
mkdir dropbox
cd dropbox
curl "https://raw.githubusercontent.com/andreafabrizi/Dropbox-Uploader/master/dropbox_uploader.sh" -o dropbox_uploader.sh
chmod 755 dropbox_uploader.sh
./dropbox_uploader.sh
Optional: Send a single file to Dropbox as a test (replace with your backup filename).

Code: Select all

/dropbox/dropbox_uploader.sh upload "/home/backup/admin.2017-03-17.tar" /

Schedule the backups to go to Dropbox daily

Make this new file:

Code: Select all

vi /usr/local/bin/send_site_backup_files_to_dropbox.sh
Add this to the file, then save it (type Esc, :wq):

Code: Select all

#!/bin/bash

#Save current date  as YYYY-MM-DD to a variable
DATE=$(date +"%Y-%m-%d")

#Loop through each file in the backup folder whose name has the current date
for X in /home/backup/*$DATE*; do
    #X is the filename with path. Remove path to get just the filename.
    NAME_NO_PATH=${X##*/}
    #Remove period and date from filename
    NEW_NAME=${NAME_NO_PATH/.$DATE/}
    #Copy the file to tmp with the new non-dated name
    cp $X /tmp/$NEW_NAME
    #Send it to Dropbox
    /dropbox/dropbox_uploader.sh -f /root/.dropbox_uploader upload "/tmp/$NEW_NAME" /
    #Delete the file from tmp
    rm -rf /tmp/$NEW_NAME
done
Note: The backup file that is uploaded to Dropbox has the date removed from its filename so that every day, the new backup will overwrite the previous backup in Dropbox. This way you don't get an accumulation of backups (e.g. admin.2017-03-17.tar, admin.2017-03-18.tar, etc...) and always only have one backup file for each user (e.g. admin.tar) in Dropbox. Using Dropbox's file version history, you can access older backups of that user. The backups on the server are unchanged (they are not renamed, and they are kept for two days as per Vesta's normal behaviour).


Optional: Run it once as a test (your backups folder must have today's backups in it).

Code: Select all

sh /usr/local/bin/send_site_backup_files_to_dropbox.sh
Verify what time Vesta does its daily backups:
  1. Log into Vesta as admin
  2. Click Cron
  3. Find “sudo /usr/local/vesta/bin/v-backup-users” and click edit
  4. Verify what time Vesta does backups (change the time if desired)
Install moreutils (needed for "ts" command used in crontab):

Code: Select all

yum install moreutils
Edit crontab:

Code: Select all

crontab -e
Add this line to crontab so your backups are sent every morning at 3:45am (or any other time you want, just make sure it is at least 30 minutes after Vesta does the backups). Change the email address at the end to whatever address you wish to get the emailed report.

Code: Select all

45 03 * * * sh /usr/local/bin/send_site_backup_files_to_dropbox.sh | ts "[\%Y-\%m-\%d \%H:\%M:\%S]" 2>&1 | tee /var/log/send_backups_to_dropbox.log | mailx -s "Report for Vesta backup files sent to Dropbox" [email protected]
Optional: You can view the log here (but this will also get emailed to as per the previous step).

Code: Select all

cat /var/log/send_backups_to_dropbox.log
That is all, your backups will now be sent to Dropbox daily at the time you specified in crontab, and you will get an email report with the results.

I have successfully used this for files up to 10 Gb in size, and I think it should work fine for even larger. You can read more about how the Dropbox Uploader script does this, and other options it offers, here: https://github.com/andreafabrizi/Dropbox-Uploader
The backup worked perfectly. But I made some changes.

After installing, I created the file /usr/local/vesta/bin/backup-dropbox.sh (permission 0755):

Code: Select all

#!/bin/bash

for entry in "$search_dir"/backup/*
do
  /dropbox/dropbox_uploader.sh upload "$entry" /backup/
done

rm -rf /backup/*
So I created a cron on the vesta panel (user root/admin) at 7 o'clock.

Code: Select all

sudo /usr/local/vesta/bin/backup-dropbox.sh

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Tue Oct 10, 2017 3:54 pm
by mehargags
Interesting... I never knew Dropbox increased their limit of single file size to be 20GB. Previously it was only 2GB per file.
https://www.dropbox.com/help/space/upload-limitations

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Mon Jan 15, 2018 4:29 pm
by ruinzaO
Friend, thanks for the great work!
I noticed that my backups were added "admin.2018-01-15_05-10-12.tar" the time, so the file is not being renamed to admin.tar in the dropbox, could you add a fix in the script?

Many thanks and I'm sorry, I'm using google.

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Mon Jan 15, 2018 10:05 pm
by vesta_mtl
ruinzaO wrote:Friend, thanks for the great work!
I noticed that my backups were added "admin.2018-01-15_05-10-12.tar" the time, so the file is not being renamed to admin.tar in the dropbox, could you add a fix in the script?

Many thanks and I'm sorry, I'm using google.
Yes, the script needed to be updated because Vesta now includes the time in the backup filename. I have updated my original post and the script should now work with the old Vesta (only yyyy-mm-dd in filename) and the new Vesta (yyyy-mm-dd_hh-mm-ss in the filename).

Let me know if you have any trouble. Your Google translation is excellent! :-)

Re: Upload Vesta user backups to Dropbox automatically daily

Posted: Tue Jan 16, 2018 11:59 am
by ruinzaO
vesta_mtl wrote:
ruinzaO wrote:Friend, thanks for the great work!
I noticed that my backups were added "admin.2018-01-15_05-10-12.tar" the time, so the file is not being renamed to admin.tar in the dropbox, could you add a fix in the script?

Many thanks and I'm sorry, I'm using google.
Yes, the script needed to be updated because Vesta now includes the time in the backup filename. I have updated my original post and the script should now work with the old Vesta (only yyyy-mm-dd in filename) and the new Vesta (yyyy-mm-dd_hh-mm-ss in the filename).

Let me know if you have any trouble. Your Google translation is excellent! :-)
Thanks for the answer!
Friend in setup this error appears.
/dropbox/dropbox_uploader.sh: line 1559: : No such file or directory

Am I doing something wrong?