FreeBSD is moving to Git
The FreeBSD Project is slowly moving away from Subversion to Git. This poses questions like how do I do a checkout, and how do I get the equivalent of Subversion’s revision numbers for use in scripts and filenames? This is not the official documentation, merely some notes for myself as I tag along.
Note, any mistakes below are my own. Please, leave a comment or send me an email if you think I’m out of my mind or worse.
stable/11 and stable/12
Users of stable/11 and stable/12 can continue using Subversion as before, and make the transition at their own pace.
Transition schedule
Here’s the tentative schedule. https://git.freebsd.org/src.git won’t exist until December 22, 2020, give or take. https://git.freebsd.org/ports.git won’t exist until early April 2021.
Draft of the FreeBSD Git transition documents
Here’s a draft of the FreeBSD Git transition documents by Warner Losh and contributors.
Browsing the repos and commit logs on the web
Head over to https://cgit.freebsd.org/ as https://svnweb.freebsd.org/ will eventually disappear.
URLs work like this:
https://cgit.freebsd.org/src/show a summary of thesrcrepo.https://cgit.freebsd.org/src/log/show the commit log for themainbranch of thesrcrepo.https://cgit.freebsd.org/src/tree/show the file directory for themainbranch of thesrcrepo.https://cgit.freebsd.org/src/log/sys/conf/newvers.shshow the commit log ofsys/conf/newvers.shin themainbranch of thesrcrepo.https://cgit.freebsd.org/src/tree/sys/conf/newvers.shshow the latest version ofsys/conf/newvers.shin themainbranch of thesrcrepo.https://cgit.freebsd.org/src/log/?h=stable/13show the commit log for thestable/13branch of thesrcrepo.https://cgit.freebsd.org/src/tree/?h=stable/13show the file directory for thestable/13branch of thesrcrepo.https://cgit.freebsd.org/src/log/sys/conf/newvers.sh?h=stable/13show the commit log ofsys/conf/newvers.shin thestable/13branch of thesrcrepo.https://cgit.freebsd.org/src/tree/sys/conf/newvers.sh?h=stable/13show the latest version ofsys/conf/newvers.shin thestable/13branch of thesrcrepo.https://cgit.freebsd.org/src/commit/?id=8a51f14a7833fd14e1f125e63a0af9d260dcd287show a specific commit in thesrcrepo.
Git, Subversion, and ViewVC
You need to set WITH_SUBVERSION_VER=LTS in /etc/make.conf and in similar files for Poudriere and Synth if you have devel/viewvc installed.
Global configuration file ~/.gitconfig
[filter "lfs"] smudge = git-lfs smudge -- %f [user] name = Trond Endrestøl email = trond.endrestol@ximalas.info [core] editor = mcedit [alias] gnlog = log --graph --pretty=format:'%Cred%h %C(green)%t %Creset %C(red)%ad %Creset-%C(yellow)%d%Creset %s %n %N %-GG' --date=short [pull] ff = only
Note, this file contains leading tabs, just like a Makefile. The tabs are probably not essential.
Cloning and checking out the 14.0-CURRENT branch
git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/src.git freebsd-src
This will download the entire repo, check out 14.0-CURRENT aka main, and make it possible to switch to other branches such as stable/12. All files end up in a directory named freebsd-src.
Cloning and checking out the 14.0-CURRENT branch on my laptop
zfs create zroot/usr/src-git git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' --depth 1 https://git.freebsd.org/src.git /usr/src-git
This should limit the initial download to the absolute minimum. When I’m ready to make the transition from Subversion to Git, I can type these commands:
zfs rename zroot/usr/src zroot/usr/src-svn zfs rename zroot/usr/src-git zroot/usr/src
Cloning and checking out the stable/12 branch
git clone -o freebsd -b stable/12 --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/src.git freebsd-src-stable-12
This will download the entire repo, check out the stable/12 branch, and make it possible to switch to other branches such as stable/13. All files end up in a directory named freebsd-src-stable-12.
Cloning and checking out the ports head branch
git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/ports.git freebsd-ports
This will download the entire repo, check out the main branch, and make it possible to switch to other branches. All files end up in a directory named freebsd-ports.
Updating the working copies
cd freebsd-src && git pull --ff-only -q && cd .. cd freebsd-src-stable-12 && git pull --ff-only -q && cd .. cd freebsd-ports && git pull --ff-only -q && cd ..
Switching from stable/12 to stable/13
cd freebsd-src-stable-12 && git checkout stable/13 cd .. && mv freebsd-src-stable-12 freebsd-src-stable-13
Switching back to CURRENT
cd freebsd-src-stable-13 && git checkout main cd .. && mv freebsd-src-stable-13 freebsd-src-main
Git’s equivalent of revision numbers
I need to distinguish each ZFS boot environment not only by date and time, but also by some unique identifier. Subversion’s global revision number has up to now served such a purpose.
CURRENT_TIMESTAMP=`date +%Y%m%d-%H%M%S`
SVN_REVISION=`svn info --show-item revision /usr/src`
NEW_SNAPSHOT="${CURRENT_TIMESTAMP}-r${SVN_REVISION}"
CURRENT_BE=`df -t zfs / | tail -n 1 | awk '{print $1}'`
BE_ROOT=`dirname ${CURRENT_BE}`
NEW_BE="${BE_ROOT}/${NEW_SNAPSHOT}"
zfs snapshot ${CURRENT_BE}@${NEW_SNAPSHOT}
zfs clone -o mountpoint=/mnt ${CURRENT_BE}@${NEW_SNAPSHOT} ${NEW_BE}
I use this construct based on commit 8a51f14a7833fd14e1f125e63a0af9d260dcd287 to remain compatible with the current kernel identification:
CURRENT_TIMESTAMP=`date +%Y%m%d-%H%M%S`
GIT_BRANCH=`git -C /usr/src rev-parse --abbrev-ref HEAD | sed 's|[/+]|-|g'`
GIT_COMMIT_COUNT=`git -C /usr/src rev-list --first-parent --count HEAD`
GIT_SHORT_COMMIT_HASH=`git -C /usr/src rev-parse --verify --short=12 HEAD`
NEW_SNAPSHOT="${CURRENT_TIMESTAMP}-${GIT_BRANCH}-n${GIT_COMMIT_COUNT}-${GIT_SHORT_COMMIT_HASH}"
CURRENT_BE=`df -t zfs / | tail -n 1 | awk '{print $1}'`
BE_ROOT=`dirname ${CURRENT_BE}`
NEW_BE="${BE_ROOT}/${NEW_SNAPSHOT}"
zfs snapshot ${CURRENT_BE}@${NEW_SNAPSHOT}
zfs clone -o mountpoint=/mnt ${CURRENT_BE}@${NEW_SNAPSHOT} ${NEW_BE}
The three Git commands above give you the branch name, a count of the available commits and a short identifier of the latest commit. Admittedly, the commit hashes don’t sort as well as Subversion’s revision numbers and they are a bit difficult to read.
I end up with ZFS filesystems and snapshots named like this:
enterprise_zroot/ROOT/20210201-152552-stable-13-local-n256298-8129345410fd enterprise_zroot/ROOT/20210201-152552-stable-13-local-n256298-8129345410fd@20210203-164224-stable-13-local-n256326-d27355aac7ca enterprise_zroot/ROOT/20210203-164224-stable-13-local-n256326-d27355aac7ca
Empty $FreeBSD$ keywords and mergemaster(8)
With empty $FreeBSD$ keywords, mergemaster(8) have nothing to compare other than the actual file contents. Previously, users of mergemaster -Fp and mergemaster -Fi only merged files when the $FreeBSD$ keyword differed. Now, we find ourselves constantly merging files such as /etc/master.passwd and /etc/group. I urge the FreeBSD project to either fix mergemaster(8) or begin manually assigning values to the $FreeBSD$ keyword for files belonging in /etc. A PR is in the works, PR 252132. See also PR 206866.
etcupdate(8) is the preferred tool. While upgrading a test system from stable/12 to stable/13, etcupdate(8) failed to handle /etc/rc.d correctly. It was eventually solved using mergemaster(8). Since etcupdate(8) is new to me, I can’t rule out pilot error on my part. This was the sequence of commands I used:
etcupdate extract etcupdate -p -F etcupdate resolve -p zfs snap ... zfs clone -o mountpoint=/mnt ... etcupdate -D /mnt -d /var/db/etcupdate -F etcupdate resolve -D /mnt -d /var/db/etcupdate etcupdate status -D /mnt -d /var/db/etcupdate
/var/db/etcupdate is a separate filesystem common to all boot environments.
After pondering the issue, I guess the use of -F is unwarranted due to $FreeBSD$ being empty and thus useless. Also, you should run the etcupdate(8) ritual on all systems before switching your source tree to stable/13 to establish a base line in /var/db/etcupdate.
I’ll try this sequence the next time I upgrade systems running main or stable/13.
etcupdate -p etcupdate resolve -p zfs snap ... zfs clone -o mountpoint=/mnt ... etcupdate -D /mnt -d /var/db/etcupdate etcupdate resolve -D /mnt -d /var/db/etcupdate etcupdate status -D /mnt -d /var/db/etcupdate
Running etcupdate status will always exit with 0. Use etcupdate status | grep -q '^ C' as the condition in a loop. See PR 254367.
etcupdate -D /mnt -d /var/db/etcupdate while etcupdate status -D /mnt -d /var/db/etcupdate | grep -q '^ C'; do etcupdate resolve -D /mnt -d /var/db/etcupdate done
Handling local changes
Kernel configuration files and local metaports are important to me. I should probably cease using CVS for my own stuff, and let Git handle things for me. I’m sure it will be a mistake not to keep extra copies of such files elsewhere.
Consider running these commands as appropriate:
git stashgit stash listgit stash applygit pull --autostash --ff-only -v
A more advanced case using my laptop as an example, would be:
git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' --depth 1 https://git.freebsd.org/src.git /usr/src cd /usr/src git checkout -b main+local cp -p /usr/src-svn/sys/amd64/conf/E590T sys/amd64/conf/E590T cp -p /usr/src-svn/sys/amd64/conf/ZFS sys/amd64/conf/ZFS git add sys/amd64/conf/E590T sys/amd64/conf/ZFS git commit
The third command created a new branch named main+local. We will use this branch for our local changes.
Ensure your cwd is /usr/src and repeat the last two commands after editing the kernel configuration file. git commit -a will never pick up new files. Get into the habit of (re)adding new and modified files prior to commit.
To update this source tree, run:
cd /usr/src git checkout main git pull --no-edit --no-ff git rebase main main+local
I imagine a similar approach can be used for the ports tree and my local metaports. Can the command sequence above be forced to run non-interactive, e.g. when run from cron(8)? Any error messages would be sent as mail, ideally leaving no processes behind, and hopefully force a human to take manual action.
For stable/13, the above sequence becomes:
cd /usr/src git checkout stable/13 git pull --no-edit --no-ff git rebase stable/13 stable/13+local
In mid-February 2021, Git gave me an error message stating I needed to run git prune. I have amended the update sequence:
cd /usr/src git prune -v git gc --auto git checkout main git pull --no-edit --no-ff git rebase main main+local
cd /usr/src git prune -v git gc --auto git checkout stable/13 git pull --no-edit --no-ff git rebase stable/13 stable/13+local
Migrating from stable/13 to stable/14 when using local branches
There are fewer hurdles to jump over if you checkout stable/14, sometime in the future, create a new local branch, stable/14+local, apply the changes between stable/13 and stable/13+local, review the changes, add the modified and new files, and finally commit the changes. All prior history of your local changes will remain in the previous branch. I did try a different approach based on git rebase, but it proved to give you too much work to do before you can even begin to see the result.
pushd /usr/src git prune -v git gc --auto git checkout stable/13 git pull --no-edit --no-ff git rebase stable/13 stable/13+local git checkout stable/14 git checkout -b stable/14+local git diff stable/13..stable/13+local | git apply # Review the modified and new files. git add ... git commit -m 'Added local changes and files.' popd
Remember to use this command sequence when updating your source tree:
git -C /usr/src prune -v git -C /usr/src gc --auto git -C /usr/src checkout stable/14 git -C /usr/src pull --no-edit --no-ff git -C /usr/src rebase stable/14 stable/14+local
Creating a mirror of the doc and src repos
zfs create -o mountpoint=/git enterprise_zdata/git zfs create enterprise_zdata/git/git.ximalas.info zfs create enterprise_zdata/git/git.ximalas.info/freebsd zfs create enterprise_zdata/git/git.ximalas.info/freebsd/doc zfs create enterprise_zdata/git/git.ximalas.info/freebsd/ports zfs create enterprise_zdata/git/git.ximalas.info/freebsd/src git clone --mirror https://git.freebsd.org/doc.git /git/git.ximalas.info/freebsd/doc git clone --mirror https://git.freebsd.org/ports.git /git/git.ximalas.info/freebsd/ports git clone --mirror https://git.freebsd.org/src.git /git/git.ximalas.info/freebsd/src
I wonder if this can be a starting point for running git daemon from inetd(8). It will be crucial to deny push and other destructive actions, only clone and pull actions should be allowed. Maybe by running git daemon as the user nobody or git_daemon, nothing bad can happen as long as the files are owned by root:wheel, and thus only the “other” access rights applies, usually r-x (dirs) and r-- (files).
I deliberately decided against adding the suffix .git to the directory names. This way, the names are similar to the ones I chose for my Subversion mirror.
/etc/inetd.conf has been amended with:
#git stream tcp46 nowait nobody /usr/local/bin/git git daemon --inetd --verbose --export-all --interpolated-path=/git/%H%D /git/git.ximalas.info/freebsd/doc /git/git.ximalas.info/freebsd/ports /git/git.ximalas.info/freebsd/src
The URLs will be:
git://git.ximalas.info/freebsd/docgit://git.ximalas.info/freebsd/portsgit://git.ximalas.info/freebsd/src
This service won’t go live until the ports repo is finalized sometime in early 2021, probably in April. I don’t encourage you to use my Git service, but you can use my setup as a template for your own private mirror. The usual disclaimers apply.
error: Could not fetch origin and error: cannot lock ref
The bare Git mirror for src at work produced the error messages error: Could not fetch origin and error: cannot lock ref. Unfortunately, I didn’t preserve the complete error messages.
Anyway, the following command was the cure:
git -C /git/git.[REDACTED].no/freebsd/src remote prune origin
One of the consumers of the above bare mirror spat out this during git -C /usr/src pull --ff-only:
remote: Enumerating objects: 77940, done. remote: Counting objects: 100% (77278/77278), done. remote: Compressing objects: 100% (19450/19450), done. remote: Total 76003 (delta 57909), reused 73920 (delta 56151), pack-reused 0 Receiving objects: 100% (76003/76003), 48.30 MiB | 20.30 MiB/s, done. Resolving deltas: 100% (57909/57909), completed with 721 local objects. From git://git.[REDACTED].no/freebsd/src 09a6fc8dbe2e..e34c3d0721bc stable/12 -> freebsd/stable/12 dd869341b1e0..4ab5c88da287 main -> freebsd/main d9d03b5409f3..90e161ec6d11 stable/11 -> freebsd/stable/11 95b7e4e0febd..923cd7e05af8 stable/13 -> freebsd/stable/13 error: cannot lock ref 'refs/remotes/freebsd/vendor/openzfs/legacy': 'refs/remotes/freebsd/vendor/openzfs' exists; cannot create 'refs/remotes/freebsd/vendor/openzfs/legacy' ! [new branch] vendor/openzfs/legacy -> freebsd/vendor/openzfs/legacy (unable to update local ref) error: cannot lock ref 'refs/remotes/freebsd/vendor/openzfs/master': 'refs/remotes/freebsd/vendor/openzfs' exists; cannot create 'refs/remotes/freebsd/vendor/openzfs/master' ! [new branch] vendor/openzfs/master -> freebsd/vendor/openzfs/master (unable to update local ref) error: cannot lock ref 'refs/remotes/freebsd/vendor/openzfs/zfs-2.1-release': 'refs/remotes/freebsd/vendor/openzfs' exists; cannot create 'refs/remotes/freebsd/vendor/openzfs/zfs-2.1-release' ! [new branch] vendor/openzfs/zfs-2.1-release -> freebsd/vendor/openzfs/zfs-2.1-release (unable to update local ref) error: some local refs could not be updated; try running 'git remote prune freebsd' to remove any old, conflicting branches
Running git -C /usr/src remote prune freebsd solved the problem.
Pruning freebsd URL: git://git.[REDACTED].no/freebsd/src * [pruned] freebsd/vendor/openzfs
git -C /usr/src pull --ff-only eventually succeeded:
From git://git.[REDACTED].no/freebsd/src * [new branch] vendor/openzfs/legacy -> freebsd/vendor/openzfs/legacy * [new branch] vendor/openzfs/master -> freebsd/vendor/openzfs/master * [new branch] vendor/openzfs/zfs-2.1-release -> freebsd/vendor/openzfs/zfs-2.1-release Updating 09a6fc8dbe2e..e34c3d0721bc Fast-forward .cirrus.yml | 1 + secure/lib/libcrypto/Version.map | 17 +++++++++++++- secure/lib/libssl/Version.map | 1 - share/man/man4/rtwn_usb.4 | 5 ++++- stand/libsa/bzipfs.c | 3 +++ stand/libsa/gzipfs.c | 3 +++ sys/dev/rtwn/usb/rtwn_usb_attach.h | 1 + sys/dev/usb/usbdevs | 1 + sys/i386/i386/machdep.c | 2 +- sys/i386/include/md_var.h | 2 +- sys/netgraph/ng_parse.c | 8 ++++--- sys/netpfil/pf/pf_table.c | 1 + sys/x86/x86/local_apic.c | 4 ---- tests/sys/netpfil/pf/killstate.sh | 13 +++++++++++ usr.sbin/etcupdate/etcupdate.8 | 25 +++++++++++++-------- usr.sbin/etcupdate/etcupdate.sh | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------------------------- usr.sbin/pciconf/cap.c | 16 +++++++++---- 17 files changed, 237 insertions(+), 104 deletions(-)
Browsing the commit logs locally
devel/tig
Don’t forget to install devel/tig as it’s a beautiful ncurses based tool for browsing the Git trees, the commit logs, the diffs, etc.
devel/tig showing commit list and a commit log entry
devel/tig showing commit list and a diff below the commit log entry
wide screen view of devel/tig showing commit list and a commit log entry
Hit the h key to get help. Hit the q key to close the help window. Hit the q key to return to the previous view, and eventually all the way back to the command prompt. Hit the Q key to exit immediately.
devel/gitui
devel/gitui is also a viable Git repo browser. There is one snag, devel/gitui can’t browse a bare mirror unlike devel/tig. Be sure to run this tool in a terminal capable of displaying modern emoji fonts. The user interface is a bit counter intuitive and without emoji fonts you have no idea of which key to hit in certain situations. You use the enter key to display the details for a commit, you then use the right arrow key to view the diff(s), but you must use the escape key to go back to main view. Why can’t we use the left arrow key, complementing the actions done by the right arrow key? Exit the program by hitting C-c.
Git equivalent of svn info
After poking around on the internet, I found a suitable replacement. I took the liberty of editing the script to make it useable on FreeBSD and in particular useful for those of us who uses local branches for our own stuff.
#!/usr/local/bin/bash # Ref: https://stackoverflow.com/questions/924574/git-alternatives-to-svn-info-that-can-be-included-in-a-build-for-traceability # author: Duane Johnson # email: duane.johnson@gmail.com # date: 2008 Jun 12 # license: MIT # # Based on discussion at http://kerneltrap.org/mailarchive/git/2007/11/12/406496 # Changelog: # 2021-04-07, Trond Endrestøl: # Added display of most recent commit on `git remote`/HEAD. # 2021-03-23, Trond Endrestøl: # Added --no-pager to nearly all git commands except git log -n 1 which already had this option. # Deactived remote branches. # Added display of cwd. # Use proper capitalisation of Git. pushd . >/dev/null # Find base of git directory while [ ! -d .git ] && [ ! `pwd` = "/" ]; do cd ..; done # Show various information about this git directory if [ -d .git ]; then echo "== WC:" pwd echo echo "== Remote URL:" git remote -v echo #echo "== Remote Branches: " #git --no-pager branch -r #echo echo "== Local Branches:" git --no-pager branch echo echo "== Configuration (.git/config)" cat .git/config echo echo "== Most Recent Commits" git --no-pager log -n 1 `git remote`/HEAD echo git --no-pager log -n 1 echo echo "Type 'git log' for more commits, or 'git show' for full commit details." else echo "Not a Git repository." fi popd >/dev/null # EOF
Optimising ZFS filesystem compression
All files in .git/objects are zlib compressed by Git. Thus, it makes sense to let .git and .git/objects be separate filesystem, turning off compression for the latter. This must be done after the initial checkout as Git expects /usr/src and /usr/ports to be empty.
cd /usr/src zfs snap zroot/usr/src@pre.git mv .git .git0 zfs create zroot/usr/src/.git zfs create -o compression=off zroot/usr/src/.git/objects mv -v .git0/objects .git mv -v .git0/* .git rm -R .git0 cd /usr/ports zfs snap zroot/usr/ports@pre.git mv .git .git0 zfs create zroot/usr/ports/.git zfs create -o compression=off zroot/usr/ports/.git/objects mv -v .git0/objects .git mv -v .git0/* .git rm -R .git0
I cheated a bit by using Midnight Commander (misc/mc) to do the actual move operations. Remove the pre.git snapshots at your own discretion.
Error message while installing devel/git 2.29.2
[5/9] Reinstalling git-2.29.2...
===> Creating groups.
Using existing group 'git_daemon'.
===> Creating users
Using existing user 'git_daemon'.
[5/9] Extracting git-2.29.2: 100%
pkg: Failed to execute lua script: [string "shell_path = pkg.prefixed_path("libexec/git-c..."]:7: attempt to index a nil value (global 'shell')
pkg: lua script failed
Huh? A Lua script failed to run. This happened on stable/12 amd64 r368505 with the ports tree at r558227.