From 8e7765ee6f6c03aa71a111bf562b543fcf0510ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20Mart=C3=AD?= Date: Wed, 23 Mar 2022 16:15:54 +0000 Subject: [PATCH] CONTRIBUTING: introduce -toolexec and add more dev tips I've explained this before, such as over chat or talks with slides, so it's time to have a proper version in our docs. It doesn't really belong in the README, because a user shouldn't have to understand these internal details. The way Go builds with -toolexec spawn sub-processes is a bit tricky, so I've attempted to demonstrate it with a vertical tree using Unicode. Each added indentation level is one more sub-process level. --- CONTRIBUTING.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 56 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 04ab5bf..0583b5f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,17 +15,71 @@ When contributing for the first time, you should also add yourself to the Just the usual `go test ./...`; many of the tests are in [testscript](https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript) under -`testdata/script/`, which allows laying out files and shell-like steps to run as +`testdata/scripts/`, which allows laying out files and shell-like steps to run as part of the test. Note that the tests do real builds, so they are quite slow; on an average laptop, `go test` can take over thirty seconds. Here are some tips: -* Use `go test -short` to skip some extra and slow sanity checks +* Use `go test -short` to skip the more expensive or thorough tests * Use `go test -run Script/foo` to just run `testdata/scripts/foo.txt` +### Integrating into Go builds + +When you run `go build`, it first loads all needed packages. +It then figures out how to build them in distinct steps, +such as the calls to `asm`, `compile`, or `link` you can see in `go build -x`. +Note how each of these toolchain tools is executed in separate processes. +For example, a small `go build` might look like the following process tree: + + go build ./main + ↳ …/compile -p project.com/library library.go + ↳ …/compile -p main main.go + ↳ …/link -o main.exe + +`garble build` boils down to calling `go build -toolexec=garble`, +which is a flag that lets us wrap the calls to each tool mentioned above. +For example, where a regular build might run `…/compile foo.go`, +`-toolexec=garble` instead runs `garble …/compile foo.go`, +which lets us obfuscate the Go source code before being compiled. + +Because of the above, `garble` gets run as multiple processes: +one top-level process that implements the CLI and the initial setup, +and one process for each compilation step tool that we transform. +For example, here's a `garble build` version of the earlier process tree: + + garble build ./main + ↳ go build -toolexec=garble ./main + ↳ garble …/compile -p project.com/library library.go + ↳ …/compile -p project.com/library library.go + ↳ garble …/compile -p main main.go + ↳ …/compile -p main main.go + ↳ garble …/link -o main.exe + ↳ …/link -o main.exe + +Go builds happen one package at a time, and so does garble's obfuscation. +This is necessary to support build caching and incremental builds. +For further build speed, packages are built in parallel whenever possible, +with each package only building once all of its dependencies are finished. + +To deduplicate work, the top-level garble process loads all packages to build, +and stores their information in a file consumed by the garble sub-processes. +Each garble sub-process also produces extra cached output of its own, +with information such as which declared names could not be obfuscated. +garble sub-processes will load the cached output files for their dependencies. + ### Development tips +To see how garble is obfuscating a build, you can use `garble -debug build`. +You can also use `-debugdir` to get a copy of the obfuscated source code. +To get finer-grained information, adding temporary debug prints is helpful. + +When investigating an issue, such as a build failure or subpar obfuscation, +it's best to reproduce the problem with the smallest build input possible. +That will help write a test case, and also make `garble -debug` more useful. +For example, if you suspect what piece of code might be causing the issue, +try moving the code to one or two new packages with very few dependencies. + To inject code into the syntax tree, don't write `go/ast` nodes by hand; you can generate them by typing Go source into tools such as [astextract](https://lu4p.github.io/astextract/).