The FreeBSD Project is slowly moving away from Subversion to Git. This poses questions like how do I do a checkout, and how do I get the equivalent of Subversion’s revision numbers for use in scripts and filenames? This is not the official documentation, merely some notes for myself as I tag along.

Note, any mistakes below are my own. Please, leave a comment or send me an email if you think I’m out of my mind or worse.

stable/11 and stable/12

Users of stable/11 and stable/12 can continue using Subversion as before, and make the transition at their own pace.

Transition schedule

Here’s the tentative schedule. https://git.freebsd.org/src.git won’t exist until December 22, 2020, give or take. https://git.freebsd.org/ports.git won’t exist until early April 2021.

Draft of the FreeBSD Git transition documents

Here’s a draft of the FreeBSD Git transition documents by Warner Losh and contributors.

Browsing the repos and commit logs on the web

Head over to https://cgit.freebsd.org/ as https://svnweb.freebsd.org/ will eventually disappear.

URLs work like this:

  • https://cgit.freebsd.org/src/ show a summary of the src repo.
  • https://cgit.freebsd.org/src/log/ show the commit log for the main branch of the src repo.
  • https://cgit.freebsd.org/src/tree/ show the file directory for the main branch of the src repo.
  • https://cgit.freebsd.org/src/log/sys/conf/newvers.sh show the commit log of sys/conf/newvers.sh in the main branch of the src repo.
  • https://cgit.freebsd.org/src/tree/sys/conf/newvers.sh show the latest version of sys/conf/newvers.sh in the main branch of the src repo.
  • https://cgit.freebsd.org/src/log/?h=stable/13 show the commit log for the stable/13 branch of the src repo.
  • https://cgit.freebsd.org/src/tree/?h=stable/13 show the file directory for the stable/13 branch of the src repo.
  • https://cgit.freebsd.org/src/log/sys/conf/newvers.sh?h=stable/13 show the commit log of sys/conf/newvers.sh in the stable/13 branch of the src repo.
  • https://cgit.freebsd.org/src/tree/sys/conf/newvers.sh?h=stable/13 show the latest version of sys/conf/newvers.sh in the stable/13 branch of the src repo.
  • https://cgit.freebsd.org/src/commit/?id=8a51f14a7833fd14e1f125e63a0af9d260dcd287 show a specific commit in the src repo.

Git, Subversion, and ViewVC

You need to set WITH_SUBVERSION_VER=LTS in /etc/make.conf and in similar files for Poudriere and Synth if you have devel/viewvc installed.

Global configuration file ~/.gitconfig

[filter "lfs"]
	smudge = git-lfs smudge -- %f
[user]
	name = Trond Endrestøl
	email = trond.endrestol@ximalas.info
[core]
	editor = mcedit
[alias]
	gnlog = log --graph --pretty=format:'%Cred%h %C(green)%t %Creset %C(red)%ad %Creset-%C(yellow)%d%Creset %s %n      %N %-GG' --date=short
[pull]
	ff = only

Note, this file contains leading tabs, just like a Makefile. The tabs are probably not essential.

Cloning and checking out the 14.0-CURRENT branch

git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/src.git freebsd-src

This will download the entire repo, check out 14.0-CURRENT aka main, and make it possible to switch to other branches such as stable/12. All files end up in a directory named freebsd-src.

Cloning and checking out the 14.0-CURRENT branch on my laptop

zfs create zroot/usr/src-git
git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' --depth 1 https://git.freebsd.org/src.git /usr/src-git

This should limit the initial download to the absolute minimum. When I’m ready to make the transition from Subversion to Git, I can type these commands:

zfs rename zroot/usr/src     zroot/usr/src-svn
zfs rename zroot/usr/src-git zroot/usr/src

Cloning and checking out the stable/12 branch

git clone -o freebsd -b stable/12 --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/src.git freebsd-src-stable-12

This will download the entire repo, check out the stable/12 branch, and make it possible to switch to other branches such as stable/13. All files end up in a directory named freebsd-src-stable-12.

Cloning and checking out the ports head branch

git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/ports.git freebsd-ports

This will download the entire repo, check out the main branch, and make it possible to switch to other branches. All files end up in a directory named freebsd-ports.

Updating the working copies

cd freebsd-src           && git pull --ff-only -q && cd ..
cd freebsd-src-stable-12 && git pull --ff-only -q && cd ..
cd freebsd-ports         && git pull --ff-only -q && cd ..

Switching from stable/12 to stable/13

cd freebsd-src-stable-12 && git checkout stable/13
cd .. && mv freebsd-src-stable-12 freebsd-src-stable-13

Switching back to CURRENT

cd freebsd-src-stable-13 && git checkout main
cd .. && mv freebsd-src-stable-13 freebsd-src-main

Git’s equivalent of revision numbers

I need to distinguish each ZFS boot environment not only by date and time, but also by some unique identifier. Subversion’s global revision number has up to now served such a purpose.

CURRENT_TIMESTAMP=`date +%Y%m%d-%H%M%S`
SVN_REVISION=`svn info --show-item revision /usr/src`
NEW_SNAPSHOT="${CURRENT_TIMESTAMP}-r${SVN_REVISION}"
CURRENT_BE=`df -t zfs / | tail -n 1 | awk '{print $1}'`
BE_ROOT=`dirname ${CURRENT_BE}`
NEW_BE="${BE_ROOT}/${NEW_SNAPSHOT}"
zfs snapshot ${CURRENT_BE}@${NEW_SNAPSHOT}
zfs clone -o mountpoint=/mnt ${CURRENT_BE}@${NEW_SNAPSHOT} ${NEW_BE}

I use this construct based on commit 8a51f14a7833fd14e1f125e63a0af9d260dcd287 to remain compatible with the current kernel identification:

CURRENT_TIMESTAMP=`date +%Y%m%d-%H%M%S`
GIT_BRANCH=`git -C /usr/src rev-parse --abbrev-ref HEAD | sed 's|[/+]|-|g'`
GIT_COMMIT_COUNT=`git -C /usr/src rev-list --first-parent --count HEAD`
GIT_SHORT_COMMIT_HASH=`git -C /usr/src rev-parse --verify --short=12 HEAD`
NEW_SNAPSHOT="${CURRENT_TIMESTAMP}-${GIT_BRANCH}-n${GIT_COMMIT_COUNT}-${GIT_SHORT_COMMIT_HASH}"
CURRENT_BE=`df -t zfs / | tail -n 1 | awk '{print $1}'`
BE_ROOT=`dirname ${CURRENT_BE}`
NEW_BE="${BE_ROOT}/${NEW_SNAPSHOT}"
zfs snapshot ${CURRENT_BE}@${NEW_SNAPSHOT}
zfs clone -o mountpoint=/mnt ${CURRENT_BE}@${NEW_SNAPSHOT} ${NEW_BE}

The three Git commands above give you the branch name, a count of the available commits and a short identifier of the latest commit. Admittedly, the commit hashes don’t sort as well as Subversion’s revision numbers and they are a bit difficult to read.

I end up with ZFS filesystems and snapshots named like this:

enterprise_zroot/ROOT/20210201-152552-stable-13-local-n256298-8129345410fd
enterprise_zroot/ROOT/20210201-152552-stable-13-local-n256298-8129345410fd@20210203-164224-stable-13-local-n256326-d27355aac7ca
enterprise_zroot/ROOT/20210203-164224-stable-13-local-n256326-d27355aac7ca

Empty $FreeBSD$ keywords and mergemaster(8)

With empty $FreeBSD$ keywords, mergemaster(8) have nothing to compare other than the actual file contents. Previously, users of mergemaster -Fp and mergemaster -Fi only merged files when the $FreeBSD$ keyword differed. Now, we find ourselves constantly merging files such as /etc/master.passwd and /etc/group. I urge the FreeBSD project to either fix mergemaster(8) or begin manually assigning values to the $FreeBSD$ keyword for files belonging in /etc. A PR is in the works, PR 252132. See also PR 206866.

etcupdate(8) is the preferred tool. While upgrading a test system from stable/12 to stable/13, etcupdate(8) failed to handle /etc/rc.d correctly. It was eventually solved using mergemaster(8). Since etcupdate(8) is new to me, I can’t rule out pilot error on my part. This was the sequence of commands I used:

etcupdate extract
etcupdate         -p -F
etcupdate resolve -p
zfs snap ...
zfs clone -o mountpoint=/mnt ...
etcupdate         -D /mnt -d /var/db/etcupdate -F
etcupdate resolve -D /mnt -d /var/db/etcupdate
etcupdate status  -D /mnt -d /var/db/etcupdate

/var/db/etcupdate is a separate filesystem common to all boot environments.

After pondering the issue, I guess the use of -F is unwarranted due to $FreeBSD$ being empty and thus useless. Also, you should run the etcupdate(8) ritual on all systems before switching your source tree to stable/13 to establish a base line in /var/db/etcupdate.

I’ll try this sequence the next time I upgrade systems running main or stable/13.

etcupdate         -p
etcupdate resolve -p
zfs snap ...
zfs clone -o mountpoint=/mnt ...
etcupdate         -D /mnt -d /var/db/etcupdate
etcupdate resolve -D /mnt -d /var/db/etcupdate
etcupdate status  -D /mnt -d /var/db/etcupdate

Running etcupdate status will always exit with 0. Use etcupdate status | grep -q '^  C' as the condition in a loop. See PR 254367.

etcupdate -D /mnt -d /var/db/etcupdate
while etcupdate status -D /mnt -d /var/db/etcupdate | grep -q '^  C'; do
  etcupdate resolve -D /mnt -d /var/db/etcupdate
done

Handling local changes

Kernel configuration files and local metaports are important to me. I should probably cease using CVS for my own stuff, and let Git handle things for me. I’m sure it will be a mistake not to keep extra copies of such files elsewhere.

Consider running these commands as appropriate:

  • git stash
  • git stash list
  • git stash apply
  • git pull --autostash --ff-only -v

A more advanced case using my laptop as an example, would be:

git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' --depth 1 https://git.freebsd.org/src.git /usr/src
cd /usr/src
git checkout -b main+local
cp -p /usr/src-svn/sys/amd64/conf/E590T sys/amd64/conf/E590T
cp -p /usr/src-svn/sys/amd64/conf/ZFS   sys/amd64/conf/ZFS
git add sys/amd64/conf/E590T sys/amd64/conf/ZFS
git commit

The third command created a new branch named main+local. We will use this branch for our local changes.

Ensure your cwd is /usr/src and repeat the last two commands after editing the kernel configuration file. git commit -a will never pick up new files. Get into the habit of (re)adding new and modified files prior to commit.

To update this source tree, run:

cd /usr/src
git checkout main
git pull --no-edit --no-ff
git rebase main main+local

I imagine a similar approach can be used for the ports tree and my local metaports. Can the command sequence above be forced to run non-interactive, e.g. when run from cron(8)? Any error messages would be sent as mail, ideally leaving no processes behind, and hopefully force a human to take manual action.

For stable/13, the above sequence becomes:

cd /usr/src
git checkout stable/13
git pull --no-edit --no-ff
git rebase stable/13 stable/13+local

In mid-February 2021, Git gave me an error message stating I needed to run git prune. I have amended the update sequence:

cd /usr/src
git prune -v
git gc --auto
git checkout main
git pull --no-edit --no-ff
git rebase main main+local
cd /usr/src
git prune -v
git gc --auto
git checkout stable/13
git pull --no-edit --no-ff
git rebase stable/13 stable/13+local

Migrating from stable/13 to stable/14 when using local branches

There are fewer hurdles to jump over if you checkout stable/14, sometime in the future, create a new local branch, stable/14+local, apply the changes between stable/13 and stable/13+local, review the changes, add the modified and new files, and finally commit the changes. All prior history of your local changes will remain in the previous branch. I did try a different approach based on git rebase, but it proved to give you too much work to do before you can even begin to see the result.

pushd /usr/src

git prune -v
git gc --auto
git checkout stable/13
git pull --no-edit --no-ff
git rebase stable/13 stable/13+local

git checkout stable/14
git checkout -b stable/14+local

git diff stable/13..stable/13+local | git apply

# Review the modified and new files.

git add ...
git commit -m 'Added local changes and files.'

popd

Remember to use this command sequence when updating your source tree:

git -C /usr/src prune -v
git -C /usr/src gc --auto
git -C /usr/src checkout stable/14
git -C /usr/src pull --no-edit --no-ff
git -C /usr/src rebase stable/14 stable/14+local

Creating a mirror of the doc and src repos

zfs create -o mountpoint=/git enterprise_zdata/git
zfs create enterprise_zdata/git/git.ximalas.info
zfs create enterprise_zdata/git/git.ximalas.info/freebsd
zfs create enterprise_zdata/git/git.ximalas.info/freebsd/doc
zfs create enterprise_zdata/git/git.ximalas.info/freebsd/ports
zfs create enterprise_zdata/git/git.ximalas.info/freebsd/src
git clone --mirror https://git.freebsd.org/doc.git /git/git.ximalas.info/freebsd/doc
git clone --mirror https://git.freebsd.org/ports.git /git/git.ximalas.info/freebsd/ports
git clone --mirror https://git.freebsd.org/src.git /git/git.ximalas.info/freebsd/src

I wonder if this can be a starting point for running git daemon from inetd(8). It will be crucial to deny push and other destructive actions, only clone and pull actions should be allowed. Maybe by running git daemon as the user nobody or git_daemon, nothing bad can happen as long as the files are owned by root:wheel, and thus only the “other” access rights applies, usually r-x (dirs) and r-- (files).

I deliberately decided against adding the suffix .git to the directory names. This way, the names are similar to the ones I chose for my Subversion mirror.

/etc/inetd.conf has been amended with:

#git	stream	tcp46	nowait	nobody	/usr/local/bin/git	git daemon --inetd --verbose --export-all --interpolated-path=/git/%H%D /git/git.ximalas.info/freebsd/doc /git/git.ximalas.info/freebsd/ports /git/git.ximalas.info/freebsd/src

The URLs will be:

  • git://git.ximalas.info/freebsd/doc
  • git://git.ximalas.info/freebsd/ports
  • git://git.ximalas.info/freebsd/src

This service won’t go live until the ports repo is finalized sometime in early 2021, probably in April. I don’t encourage you to use my Git service, but you can use my setup as a template for your own private mirror. The usual disclaimers apply.

error: Could not fetch origin and error: cannot lock ref

The bare Git mirror for src at work produced the error messages error: Could not fetch origin and error: cannot lock ref. Unfortunately, I didn’t preserve the complete error messages.

Anyway, the following command was the cure:

git -C /git/git.[REDACTED].no/freebsd/src remote prune origin

One of the consumers of the above bare mirror spat out this during git -C /usr/src pull --ff-only:

remote: Enumerating objects: 77940, done.
remote: Counting objects: 100% (77278/77278), done.
remote: Compressing objects: 100% (19450/19450), done.
remote: Total 76003 (delta 57909), reused 73920 (delta 56151), pack-reused 0
Receiving objects: 100% (76003/76003), 48.30 MiB | 20.30 MiB/s, done.
Resolving deltas: 100% (57909/57909), completed with 721 local objects.
From git://git.[REDACTED].no/freebsd/src
   09a6fc8dbe2e..e34c3d0721bc  stable/12                      -> freebsd/stable/12
   dd869341b1e0..4ab5c88da287  main                           -> freebsd/main
   d9d03b5409f3..90e161ec6d11  stable/11                      -> freebsd/stable/11
   95b7e4e0febd..923cd7e05af8  stable/13                      -> freebsd/stable/13
error: cannot lock ref 'refs/remotes/freebsd/vendor/openzfs/legacy': 'refs/remotes/freebsd/vendor/openzfs' exists; cannot create 'refs/remotes/freebsd/vendor/openzfs/legacy'
 ! [new branch]                vendor/openzfs/legacy          -> freebsd/vendor/openzfs/legacy  (unable to update local ref)
error: cannot lock ref 'refs/remotes/freebsd/vendor/openzfs/master': 'refs/remotes/freebsd/vendor/openzfs' exists; cannot create 'refs/remotes/freebsd/vendor/openzfs/master'
 ! [new branch]                vendor/openzfs/master          -> freebsd/vendor/openzfs/master  (unable to update local ref)
error: cannot lock ref 'refs/remotes/freebsd/vendor/openzfs/zfs-2.1-release': 'refs/remotes/freebsd/vendor/openzfs' exists; cannot create 'refs/remotes/freebsd/vendor/openzfs/zfs-2.1-release'
 ! [new branch]                vendor/openzfs/zfs-2.1-release -> freebsd/vendor/openzfs/zfs-2.1-release  (unable to update local ref)
error: some local refs could not be updated; try running
 'git remote prune freebsd' to remove any old, conflicting branches

Running git -C /usr/src remote prune freebsd solved the problem.

Pruning freebsd
URL: git://git.[REDACTED].no/freebsd/src
 * [pruned] freebsd/vendor/openzfs

git -C /usr/src pull --ff-only eventually succeeded:

From git://git.[REDACTED].no/freebsd/src
 * [new branch]                vendor/openzfs/legacy          -> freebsd/vendor/openzfs/legacy
 * [new branch]                vendor/openzfs/master          -> freebsd/vendor/openzfs/master
 * [new branch]                vendor/openzfs/zfs-2.1-release -> freebsd/vendor/openzfs/zfs-2.1-release
Updating 09a6fc8dbe2e..e34c3d0721bc
Fast-forward
 .cirrus.yml                        |   1 +
 secure/lib/libcrypto/Version.map   |  17 +++++++++++++-
 secure/lib/libssl/Version.map      |   1 -
 share/man/man4/rtwn_usb.4          |   5 ++++-
 stand/libsa/bzipfs.c               |   3 +++
 stand/libsa/gzipfs.c               |   3 +++
 sys/dev/rtwn/usb/rtwn_usb_attach.h |   1 +
 sys/dev/usb/usbdevs                |   1 +
 sys/i386/i386/machdep.c            |   2 +-
 sys/i386/include/md_var.h          |   2 +-
 sys/netgraph/ng_parse.c            |   8 ++++---
 sys/netpfil/pf/pf_table.c          |   1 +
 sys/x86/x86/local_apic.c           |   4 ----
 tests/sys/netpfil/pf/killstate.sh  |  13 +++++++++++
 usr.sbin/etcupdate/etcupdate.8     |  25 +++++++++++++--------
 usr.sbin/etcupdate/etcupdate.sh    | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------------------------------------
 usr.sbin/pciconf/cap.c             |  16 +++++++++----
 17 files changed, 237 insertions(+), 104 deletions(-)

Browsing the commit logs locally

devel/tig

Don’t forget to install devel/tig as it’s a beautiful ncurses based tool for browsing the Git trees, the commit logs, the diffs, etc.

devel/tig showing commit list and a commit log entry

devel/tig showing commit list and a commit log entry

devel/tig showing commit list and a diff below the commit log entry

devel/tig showing commit list and a diff below the commit log entry

wide screen view of devel/tig showing commit list and a commit log entry

wide screen view of devel/tig showing commit list and a commit log entry

Hit the h key to get help. Hit the q key to close the help window. Hit the q key to return to the previous view, and eventually all the way back to the command prompt. Hit the Q key to exit immediately.

devel/gitui

devel/gitui is also a viable Git repo browser. There is one snag, devel/gitui can’t browse a bare mirror unlike devel/tig. Be sure to run this tool in a terminal capable of displaying modern emoji fonts. The user interface is a bit counter intuitive and without emoji fonts you have no idea of which key to hit in certain situations. You use the enter key to display the details for a commit, you then use the right arrow key to view the diff(s), but you must use the escape key to go back to main view. Why can’t we use the left arrow key, complementing the actions done by the right arrow key? Exit the program by hitting C-c.

Git equivalent of svn info

After poking around on the internet, I found a suitable replacement. I took the liberty of editing the script to make it useable on FreeBSD and in particular useful for those of us who uses local branches for our own stuff.

#!/usr/local/bin/bash

# Ref: https://stackoverflow.com/questions/924574/git-alternatives-to-svn-info-that-can-be-included-in-a-build-for-traceability

# author: Duane Johnson
# email: duane.johnson@gmail.com
# date: 2008 Jun 12
# license: MIT
#
# Based on discussion at http://kerneltrap.org/mailarchive/git/2007/11/12/406496

# Changelog:

# 2021-04-07, Trond Endrestøl:
#   Added display of most recent commit on `git remote`/HEAD.

# 2021-03-23, Trond Endrestøl:
#   Added --no-pager to nearly all git commands except git log -n 1 which already had this option.
#   Deactived remote branches.
#   Added display of cwd.
#   Use proper capitalisation of Git.

pushd . >/dev/null

# Find base of git directory
while [ ! -d .git ] && [ ! `pwd` = "/" ]; do cd ..; done

# Show various information about this git directory
if [ -d .git ]; then
  echo "== WC:"
  pwd
  echo

  echo "== Remote URL:"
  git remote -v
  echo

  #echo "== Remote Branches: "
  #git --no-pager branch -r
  #echo

  echo "== Local Branches:"
  git --no-pager branch
  echo

  echo "== Configuration (.git/config)"
  cat .git/config
  echo

  echo "== Most Recent Commits"
  git --no-pager log -n 1 `git remote`/HEAD
  echo
  git --no-pager log -n 1
  echo

  echo "Type 'git log' for more commits, or 'git show' for full commit details."
else
  echo "Not a Git repository."
fi

popd >/dev/null

# EOF

Optimising ZFS filesystem compression

All files in .git/objects are zlib compressed by Git. Thus, it makes sense to let .git and .git/objects be separate filesystem, turning off compression for the latter. This must be done after the initial checkout as Git expects /usr/src and /usr/ports to be empty.

cd /usr/src
zfs snap zroot/usr/src@pre.git
mv .git .git0
zfs create zroot/usr/src/.git
zfs create -o compression=off zroot/usr/src/.git/objects
mv -v .git0/objects .git
mv -v .git0/* .git
rm -R .git0

cd /usr/ports
zfs snap zroot/usr/ports@pre.git
mv .git .git0
zfs create zroot/usr/ports/.git
zfs create -o compression=off zroot/usr/ports/.git/objects
mv -v .git0/objects .git
mv -v .git0/* .git
rm -R .git0

I cheated a bit by using Midnight Commander (misc/mc) to do the actual move operations. Remove the pre.git snapshots at your own discretion.

Error message while installing devel/git 2.29.2

[5/9] Reinstalling git-2.29.2...
===> Creating groups.
Using existing group 'git_daemon'.
===> Creating users
Using existing user 'git_daemon'.
[5/9] Extracting git-2.29.2: 100%
pkg: Failed to execute lua script: [string "shell_path = pkg.prefixed_path("libexec/git-c..."]:7: attempt to index a nil value (global 'shell')
pkg: lua script failed

Huh? A Lua script failed to run. This happened on stable/12 amd64 r368505 with the ports tree at r558227.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>