Making your own FreeBSD Subversion repository mirror
Not long ago it was announced that the FreeBSD ports tree will cease exporting its Subversion repository to CVS, and subsequently any use of CVSup for updating the ports tree will be discontinued by February 28th 2013.
FreeBSD’s main source tree repository has been served by Subversion since late May 2008 with every commit done in the Subversion repository being exported to the old CVS repository, but no date has been announced when that Subversion to CVS transfer will be shut down. Stay tuned for more information.
Nonetheless, this is a Good Time™ to begin the transition from CVSup to Subversion once and for all. And why not set up your own FreeBSD Subversion repository mirror for both the main source tree and the ports tree well ahead of the transition?
For more official information, you should consult the FreeBSD’s Committer’s Guide chapter on Subversion and the FreeBSD Handbook‘s list of official FreeBSD Subversion repository mirrors.
On one of my own computers I have previously installed and set up Subversion for serving the repositories I chose to share with the rest of the world. All Subversion repositories (usually) live in /home/svn/svnroot. In this case /home/svn/svnroot is a separate ZFS file system. The same is true for /home/svn. I don’t feel it’s necessary for me to backup the FreeBSD Subversion mirrors, so I chose to create a separate ZFS file system for these two FreeBSD repository mirrors, namely /home/svn/svnroot/freebsd.
Here’s the command I used:
zfs create -o compression=gzip-9 enterprise_zdata/home/svn/svnroot/freebsd
What happens next is dependent on how much time and bandwidth you are willing to spend. You could start off by using a seed file available from ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/ and synchronize the missing revisions afterwards, or you could start fresh by downloading each and every revision. I chose the latter which involves a lot more work, testing my patience.
Next, I logged in as the svn user and prepared two skeleton Subversion repository on which we will dump the contents from one of the official FreeBSD Subversion mirrors.
cd ~svn/svnroot/freebsd svnadmin create base svnadmin create ports
Some surgery is necessary before we can proceed with svnsync.
First, allow for anonymous read access by editing the conf/svnserve.conf file. The line:
# anon-access = read
in the [general] section, usually at line no. 19, should read:
anon-access = read
Copy the edited svnserve.conf file to the ports repository:
cp -p svnserve.conf ../../ports/conf/svnserve.conf
Second, and most important(!), the pre-commit hook for revprop changes must exist and be executable. Thus, copy the hooks/pre-revprop-change.tmpl file to hooks/pre-revprop-change, add execute permissions, and edit the script to exit with exit status zero.
cd ~svn/svnroot/freebsd/base/hooks cp -p pre-revprop-change.tmpl pre-revprop-change chmod a+x pre-revprop-change
The first few lines of the pre-revprop-change file should read:
#!/bin/sh exit 0 # PRE-REVPROP-CHANGE HOOK # # The pre-revprop-change hook is invoked before a revision property # is added, modified or deleted. Subversion runs this hook by invoking # a program (script, executable, binary, etc.) named 'pre-revprop-change' # (for which this file is a template), with the following ordered # arguments:
Copy the edited pre-revprop-change file to the hooks directory for the ports repository mirror:
cp -p pre-revprop-change ../../ports/hooks/pre-revprop-change
Now, we need to make sure the repositories are ready for receiving the contents from one of the official FreeBSD Subversion mirrors. I chose to use the svn0.us-east.freebsd.org mirror, but you may opt to use the svn0.us-west.freebsd.org mirror, or the svn0.eu.freebsd.org mirror. It is also possible to use the main FreeBSD Subversion repository at svn.freebsd.org, but that may change in the future.
svnsync init file:///home/svn/svnroot/freebsd/base svn://svn0.us-east.freebsd.org/base svnsync init file:///home/svn/svnroot/freebsd/ports svn://svn0.us-east.freebsd.org/ports
The next two commands are both time consuming and bandwidth hungry when executed for the first time:
svnsync sync file:///home/svn/svnroot/freebsd/base svnsync sync file:///home/svn/svnroot/freebsd/ports
In my case, the first command ran for 39 hours and 20 minutes. The second command ran for 55 hours and 38 minutes.
If you at some time later want to change which mirror you are updating your own mirror from, then study these commands:
svn propset svn:sync-from-url --revprop -r 0 svn://svn0.us-west.freebsd.org/base file:///home/svn/svnroot/freebsd/base svn propset svn:sync-from-url --revprop -r 0 svn://svn0.us-west.freebsd.org/ports file:///home/svn/svnroot/freebsd/ports
To make it easy for people to simply migrate from using one of the official mirrors to my mirror, we need to mimick the UUIDs of the main FreeBSD Subversion repositories.
svnadmin setuuid /home/svn/svnroot/freebsd/base ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f svnadmin setuuid /home/svn/svnroot/freebsd/ports 35697150-7ecd-e111-bb59-0022644237b5
The two commands above allows every user to simply execute the following commands:
cd /usr/src svn relocate svn://svn.ximalas.info/freebsd/base/stable/9 cd /usr/ports svn relocate svn://svn.ximalas.info/freebsd/ports/head
I’m assuming you are already a Subversion user and that you are following the stable/9 line. As a result you are now syncronizing your working copies from my FreeBSD Subversion mirror.
This would be no good if my mirror is not synchronizing itself with the chosen official mirror, so let’s set up a crontab file for the svn user enabling synchronization every two hours.
SHELL=/bin/sh PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin HOME=/home/svn MAILTO=svn # #minute hour mday month wday command # 0 */2 * * * svnsync sync -q file:///home/svn/svnroot/freebsd/base 0 */2 * * * svnsync sync -q file:///home/svn/svnroot/freebsd/ports
This file is expected to have tabs between each field, mind you.
Save the file as .crontab in the home directory of your svn user, and let the cron utility know what to do using the command:
crontab .crontab
If you are new to FreeBSD, you may choose to checkout RELENG_9 aka stable/9, and the ports collection, by using these commands, tailor them to suit your needs:
svn co svn://svn.ximalas.info/freebsd/base/stable/9 /usr/src svn co svn://svn.ximalas.info/freebsd/ports/head /usr/ports
On the other hand, you might be better off using one of the official FreeBSD Subversion mirrors, svn0.us-west.freebsd.org, or svn0.us-east.freebsd.org, or even svn.freebsd.org. It’s entirely up to you.
Here are some select items from ZFS’ statistics, i.e. a command like zfs get all enterprise_zdata/home/svn/svnroot/freebsd, updated on 2013-03-24:
PROPERTY |
VALUE |
|---|---|
used |
14.2G |
compressratio |
2.76x |
recordsize |
128K |
checksum |
fletcher4 |
compression |
gzip-9 |
exec |
on |
setuid |
off |
logicalused |
23.2G |
The base repository mirror consumes roughly 5.1 GiB, while the ports repository consumes roughly 8.8 GiB, both as reported by the du command.
The reason for these weird numbers stems from the fact that I use hard drives with 4K disc block sizes and have enabled compression. The lesson is simply don’t do that.
I whipped up this small shell script to create my own seed files, much like the seed files available at ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/. In effect the seed files are an online backup of my repository mirror.
#!/bin/sh
# Shell script for creating seed files for the FreeBSD Subversion repository mirror.
# Copyright © 2012, Trond Endrestøl <Trond.Endrestol@ximalas.info>
# All rights reserved.
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
FREEBSD_SVN_REPO_MIRROR_DIR="/home/svn/svnroot/freebsd"
FREEBSD_SVN_REPO_MIRROR_FS="enterprise_zdata/home/svn/svnroot/freebsd"
FREEBSD_SVN_REPO_SEEDFILE_DIR="/home/svn/tmp"
LOCAL_MIRROR_HOSTNAME="svn.ximalas.info"
PATH=/bin:/sbin:/usr/bin:/usr/local/bin
export PATH
SNAPSHOTNAME=`date +%Y%m%dT%H%M%S`
SNAPSHOT="${FREEBSD_SVN_REPO_MIRROR_FS}@${SNAPSHOTNAME}"
zfs snapshot ${SNAPSHOT}
cd ${FREEBSD_SVN_REPO_MIRROR_DIR}/.zfs/snapshot/${SNAPSHOTNAME}
# Samples from manual execution:
# -rw-r--r-- 1 svn svn - 1,2G 11 sep 19:01 svnmirror-base-r240363.svn.ximalas.info.tar.xz
# -rw-r--r-- 1 svn svn - 714M 11 sep 22:26 svnmirror-ports-r301235.svn.ximalas.info.tar.xz
BASE_REVISION_NUMBER=`tail -2 base/db/revprops/0/0 | head -1`
PORTS_REVISION_NUMBER=`tail -2 ports/db/revprops/0/0 | head -1`
gtar cJf ${FREEBSD_SVN_REPO_SEEDFILE_DIR}/svnmirror-base-r${BASE_REVISION_NUMBER}.${LOCAL_MIRROR_HOSTNAME}.tar.xz base
gtar cJf ${FREEBSD_SVN_REPO_SEEDFILE_DIR}/svnmirror-ports-r${PORTS_REVISION_NUMBER}.${LOCAL_MIRROR_HOSTNAME}.tar.xz ports
cd /
zfs destroy ${SNAPSHOT}
exit 0
Update 2013-03-24
Those who are still using CVS/CVSup for updating their ports tree will see messages like these:
"/usr/ports/Mk/bsd.port.mk", line 4: warning: ACTION REQUIRED "/usr/ports/Mk/bsd.port.mk", line 5: warning: You are using a ports file that originated from CVS!! "/usr/ports/Mk/bsd.port.mk", line 6: warning: The FreeBSD project has switched from CVS to SubVersion. "/usr/ports/Mk/bsd.port.mk", line 7: warning: This CVS repository is NO LONGER UPDATED! If you see this "/usr/ports/Mk/bsd.port.mk", line 8: warning: message then your tree is STALE and you need to follow "/usr/ports/Mk/bsd.port.mk", line 9: warning: the update instructions to receive any more updates. "/usr/ports/Mk/bsd.port.mk", line 10: warning: Original announcement: "/usr/ports/Mk/bsd.port.mk", line 11: warning: http://lists.freebsd.org/pipermail/freebsd-ports/2012-September/078099.html "/usr/ports/Mk/bsd.port.mk", line 12: warning: Reminder: "/usr/ports/Mk/bsd.port.mk", line 13: warning: http://lists.freebsd.org/pipermail/freebsd-announce/2013-January/001451.html "/usr/ports/Mk/bsd.port.mk", line 14: warning: UPDATE INSTRUCTIONS: "/usr/ports/Mk/bsd.port.mk", line 15: warning: http://wiki.freebsd.org/CvsIsDeprecated
Switch to the FreeBSD ports Subversion repository using either devel/subversion or net/svnup. The latter is recommended if you’re short on disk space or don’t want to waste additional disk space due to the .svn control directory.
To access the FreeBSD ports Subversion repository mirror on my server using (a recent version of) svnup, you would use a command like this one:
svnup current -b freebsd/ports/head -f -h svn.ximalas.info -l /usr/ports -p svn -v 2
svnup is a bit too primitive since it deletes files and directories it should leave in place:
####### Fetching revision: 315115 - /usr/ports/INDEX-9.db - /usr/ports/INDEX-9 - /usr/ports/distfiles/Net-SSLeay-1.54.tar.gz - /usr/ports/distfiles/IO-Socket-SSL-1.84.tar.gz - /usr/ports/distfiles/Net-HTTP-6.06.tar.gz - /usr/ports/distfiles/LWP-Protocol-https-6.03.tar.gz - /usr/ports/distfiles/Crypt-SSLeay-0.64.tar.gz - /usr/ports/distfiles/libwww-perl-6.05.tar.gz - /usr/ports/distfiles/Mail-SpamAssassin-3.3.2.tar.gz - /usr/ports/distfiles/spamass-milter-0.3.2.tar.gz - /usr/ports/distfiles
You can make up for this deficiency by creating separate directories for distfiles and packages, and optionally a directory for workdirs, like this:
mkdir /usr/ports-distfiles mkdir /usr/ports-packages mkdir /usr/ports-workdirs
Next, add the following variables to the /etc/make.conf file:
DISTDIR=/usr/ports-distfiles PACKAGES=/usr/ports-packages WRKDIRPREFIX=/usr/ports-workdirs
If you keep local stuff in /usr/ports/local, then you’d better store that stuff in CVS or in some other SCM system and be prepared to check out those files after each svnup run.
While the preceding paragraphs on how to run snvup are sound advice, the latest version of svnup, currently at 0.99, is a great improvement over its predecessors, and it no longer deletes excess files not originating from the Subversion repositories unless told to do so.