Issue53

classification
Title Set up backup regime
Type defect Module tracker
Severity data loss Platform N/A
Keywords workflow Nosy List stephen
explanation
process
These controls should only be changed by committers and tracker administrators.
Status assigned   Reason
Superseder  
Priority normal   Assigned To

Created on 2008-01-18.07:02:00 by stephen, last changed 2013-09-02.06:27:54 by stephen.

Messages
msg2054 [hidden] ([hidden]) Date: 2010-01-10.02:20:28
  Message-ID: <1263090028.39.0.41865740478.issue53@xemacs.org>
Hi Stephen,

From msg1886, it seemed that you've tar, cp or cpio a git repository. Is
that operation safe for git? In my experience I'd break my hg repository
this way (can be checked by run 'hg verify' in copied repository)
sometime ago. 

BR,
FKtPp
msg1948 [hidden] ([hidden]) Date: 2009-10-25.18:47:14
Cron job is working.
msg1887 [hidden] ([hidden]) Date: 2009-10-03.13:11:51
TODO:
1. Check that the cron job is running.
2. Set up automatic offsite copying.
3. Test that the backup can actually be restored.
msg1886 [hidden] ([hidden]) Date: 2009-10-03.13:00:39
The script to do a full backup is run as a cron job daily at 4am Japan time.
That's 7pm in London and 11am in San Francisco.  This means interruption of
service for about 15 seconds at those times.

The current backup script as of 2009-10-03:

TIMESTAMP=`date +%Y%m%d.%H%M%S`
BAKDIR=/var/www/trackers/db-backups
echo TIMESTAMP = $TIMESTAMP  BAKDIR=$BAKDIR
if test -e $BAKDIR -a ! -d $BAKDIR; then
  echo "$BAKDIR exists but is not a directory"
  exit -1
elif test ! -e $BAKDIR; then
  mkdir -p $BAKDIR
  chown root:postgres $BAKDIR
  chmod g+w $BAKDIR
# we should check ownership here ... rewrite in python?
fi
/etc/init.d/exim4 stop
/etc/init.d/apache2 stop
echo su -c "pg_dumpall > $BAKDIR/$TIMESTAMP-pgdb.out" postgres
su -c "pg_dumpall > $BAKDIR/$TIMESTAMP-pgdb.out" postgres
cd /var/www/trackers
echo tar czf $BAKDIR/$TIMESTAMP-xemacs.tar.gz xemacs 
tar czf $BAKDIR/$TIMESTAMP-xemacs.tar.gz xemacs
/etc/init.d/apache2 start
/etc/init.d/exim4 start

Some statistics on the dump process:

Backup timing:
real	0m12.407s
user	0m9.041s
sys	0m1.108s

NOTE:
Space use is a little scary at first sight, but look closely:
it's mostly in the logs.

Backup data ls -l:
-rw-r--r-- 1 postgres postgres  1335680 2009-10-03 21:44 20091003.214417-pgdb.out
-rw-r--r-- 1 root     root     31571177 2009-10-03 21:44 20091003.214417-xemacs.tar.gz

XEmacs tracker du (PostgreSQL specific, includes non-XEmacs data):

4	/var/lib/postgresql/8.4/main/pg_tblspc
12	/var/lib/postgresql/8.4/main/pg_clog
5484	/var/lib/postgresql/8.4/main/base/11563
5592	/var/lib/postgresql/8.4/main/base/11564
5484	/var/lib/postgresql/8.4/main/base/1
16564	/var/lib/postgresql/8.4/main/base
460	/var/lib/postgresql/8.4/main/global
12	/var/lib/postgresql/8.4/main/pg_subtrans
12	/var/lib/postgresql/8.4/main/pg_multixact/offsets
12	/var/lib/postgresql/8.4/main/pg_multixact/members
28	/var/lib/postgresql/8.4/main/pg_multixact
4	/var/lib/postgresql/8.4/main/pg_xlog/archive_status
16412	/var/lib/postgresql/8.4/main/pg_xlog
12	/var/lib/postgresql/8.4/main/pg_stat_tmp
4	/var/lib/postgresql/8.4/main/pg_twophase
33524	/var/lib/postgresql/8.4/main/

XEmacs tracker du (Roundup specific)
8	xemacs/extensions
193560	xemacs/logs
28	xemacs/detectors
10340	xemacs/db/files
10736	xemacs/db/text-index
21084	xemacs/db
496	xemacs/html
4	xemacs/static
60	xemacs/.git/logs
36	xemacs/.git/refs
8	xemacs/.git/info
44	xemacs/.git/hooks
4288	xemacs/.git/objects
4468	xemacs/.git
256	xemacs/backup/2008-01-09
484	xemacs/backup/2008-01-18
400	xemacs/backup/2008-01-16
4368	xemacs/backup/2008-01-22
5512	xemacs/backup
238420	xemacs
msg1885 [hidden] ([hidden]) Date: 2009-10-03.10:42:49
Something like this script should do it (execute as root)

TIMESTAMP=`date %Y%m%d%H%M%S`
/etc/init.d/exim4 stop
/etc/init.d/apache2 stop
pg_dumpall > /var/www/trackers/db-backup/${TIMESTAMP}-pgdb.out
cd /var/www/trackers
tar czf db-backup DataAnalysis dynamics test xemacs zemi
/etc/init.d/apache2 start
/etc/init.d/exim4 start

From the Roundup Administrator's Guide:

Tracker Backup

The roundup-admin import and export commands are not recommended for 
performing backup.

Optionally stop the web and email frontends and to copy the contents of 
the tracker home directory to some other place using standard backup 
tools. This means using pg_dump to take a snapshot of your Postgres 
backend database, for example. A simple copy of the tracker home (and 
files storage area if you’ve configured it to be elsewhere) will then 
complete the backup.
msg547 [hidden] ([hidden]) Date: 2008-02-08.09:39:13
Current setup with Mailman archiving xemacs-tracker is helpful,
but not complete.  Very little comes in through mail.
History
Date User Action Args
2013-09-02 06:27:54stephensetstatus: closed -> assigned
priority: urgent -> normal
2013-08-24 00:32:21iWontFixYourBugsetstatus: committed -> closed
2010-01-10 02:20:28FKtPpsetmessages: + msg2054
2009-10-25 18:47:14stephensetmessages: + msg1948
2009-10-03 13:11:51stephensetmessages: + msg1887
2009-10-03 13:00:40stephensetstatus: assigned -> committed
messages: + msg1886
2009-10-03 10:42:49stephensetmessages: + msg1885
2009-10-03 09:11:59stephensetstatus: in progress -> assigned
2008-02-08 09:39:13stephensetmessages: + msg547
2008-01-22 18:24:11stephensetkeyword: + workflow
2008-01-19 08:14:15stephensetstatus: new -> in progress
2008-01-18 07:02:00stephencreate