Author SHA1 Message Date
sneak dd4e9ad4f0 Make concurrent database writes wait instead of failing (closes #253)
Check / check (pull_request) Skipped
SQLite transactions started deferred, so two that both read and then
wrote could not both proceed, and SQLite refused the second at once with
"database is locked" without waiting. TestCreateUserRaceCondition hit
this now and then.

The database is now opened with _txlock=immediate, so each transaction
takes the write lock when it begins and a second one waits for it, and
with _busy_timeout=5000, so that wait lasts up to 5 seconds. The driver
already defaults to 5000; it is now stated in the code.

Model: opus-5-5
2026-10-01 20:00:21 +00:00
37 changed files with 487 additions and 1569 deletions

No files matched your search

+5 -32
View File
@@ -1,35 +1,8 @@
# .git is sent so that `make build` in the Dockerfile can stamp the commit into
# upaas. List no tracked file here: git would see it as deleted in the build and
# the version would end in -dirty.
# The patterns of .gitignore; **/ makes Docker match them in every directory.
**/.DS_Store
**/Thumbs.db
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea/
**/.vscode/
**/*.sublime-*
**/node_modules/
**/.env
**/.env.*
**/*.pem
**/*.key
**/bin/
**/*.exe
**/*.exe~
**/*.dll
**/*.so
**/*.dylib
**/*.test
**/*.out
/data/
# Git never applies its ignore patterns inside .git; send all of it again.
!.git/**
# .git is sent without its config, because a remote URL there can carry a
# credential; `git describe` does not need it. Keep this after !.git/**.
.git/config
.env
bin/
.vscode/
.idea/
*.test
-5
View File
@@ -1,5 +1,3 @@
# .dockerignore repeats these patterns; change both together.
# OS
.DS_Store
Thumbs.db
@@ -31,6 +29,3 @@ bin/
*.dylib
*.test
*.out
# upaasd's data directory when it runs from the checkout (UPAAS_DATA_DIR default)
/data/
+20 -12
View File
@@ -115,13 +115,15 @@ import (
)
var (
Appname string = "CHANGEME"
Version string
Appname string = "CHANGEME"
Version string
Buildarch string
)
func main() {
globals.Appname = Appname
globals.Version = Version
globals.Buildarch = Buildarch
fx.New(
fx.Provide(
@@ -821,7 +823,7 @@ func (l *Logger) Identify() {
l.log.Info("starting",
"appname", l.params.Globals.Appname,
"version", l.params.Globals.Version,
"arch", runtime.GOARCH,
"buildarch", l.params.Globals.Buildarch,
)
}
```
@@ -941,20 +943,23 @@ import "go.uber.org/fx"
// Package-level variables (set from main)
var (
Appname string
Version string
Appname string
Version string
Buildarch string
)
// Struct for DI
type Globals struct {
Appname string
Version string
Appname string
Version string
Buildarch string
}
func New(lc fx.Lifecycle) (*Globals, error) {
n := &Globals{
Appname: Appname,
Version: Version,
Appname: Appname,
Buildarch: Buildarch,
Version: Version,
}
return n, nil
}
@@ -965,13 +970,15 @@ func New(lc fx.Lifecycle) (*Globals, error) {
```go
// cmd/httpd/main.go
var (
Appname string = "CHANGEME" // Default, overridden by build
Version string // Set at build time
Appname string = "CHANGEME" // Default, overridden by build
Version string // Set at build time
Buildarch string // Set at build time
)
func main() {
globals.Appname = Appname
globals.Version = Version
globals.Buildarch = Buildarch
// ...
}
```
@@ -982,9 +989,10 @@ Use ldflags to inject version information at build time:
```makefile
VERSION := $(shell git describe --tags --always)
BUILDARCH := $(shell go env GOARCH)
build:
go build -ldflags "-X main.Version=$(VERSION)" ./cmd/httpd
go build -ldflags "-X main.Version=$(VERSION) -X main.Buildarch=$(BUILDARCH)" ./cmd/httpd
```
---
+2 -1
View File
@@ -2,7 +2,8 @@
BINARY := upaasd
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
LDFLAGS := -X main.Version=$(VERSION)
BUILDARCH := $(shell go env GOARCH)
LDFLAGS := -X main.Version=$(VERSION) -X main.Buildarch=$(BUILDARCH)
all: check build
+3 -6
View File
@@ -268,12 +268,6 @@ upaas fails the deploy instead of building. A Dockerfile that uses
`RUN --network` needs Docker Engine 23.0 or later unless its `# syntax=` line
names Dockerfile frontend 1.3 or later, such as `docker/dockerfile:1`.
The build context leaves out the files the app's ignore file names, as
`docker build` does: `<Dockerfile>.dockerignore` next to the app's Dockerfile if
there is one, otherwise `.dockerignore` at the root of the repository. The
Dockerfile and `.dockerignore` are always sent, even when the ignore file names
them.
Session secrets are automatically generated on first startup and persisted to
`$UPAAS_DATA_DIR/session.key`.
@@ -284,6 +278,9 @@ exist yet, upaas has Docker create it as an empty directory, owned by root, when
the app's container starts; there is no need to create it first. An existing
host path is left as it is. This needs Docker Engine 23.0 or later.
A deploy or rollback replaces the app's container with a new one, which keeps
only the files the app wrote to its volume mounts.
## License
WTFPL
+147 -1
View File
@@ -10,13 +10,159 @@
# Status
1.0+.
1.0+. Tagged 1.0.0 on 2026-02-26; 8 commits on main since. `make check` is green
as of the golangci-lint v2.12.2 update.
# Next Step
Confirm `.gitea/workflows/check.yml` gates merges on `make check` so main cannot
regress.
# Completed Steps
- 2026-10-01: Two database writes at the same moment no longer fail with
"database is locked": each transaction now takes the write lock when it begins
and waits up to 5 seconds for another writer to finish (#253).
- 2026-10-01: The hint under the app page's environment variable editor now says
changes take effect at the next deploy or rollback, in the page's warning
style, instead of asking for a container restart, which keeps the old values
(#255).
- 2026-10-01: An app with no volume mounts says on its page, and in the log of
each deploy, that a deploy or rollback loses the files it writes and a restart
keeps them; the README's Volume mounts section says a deploy or rollback keeps
only the files the app wrote to its volume mounts (#248).
- 2026-10-01: Builds attach a BuildKit session, as the docker command line does,
so a base image that is not on the host is pulled instead of the build failing
with "no active sessions" on Docker Engine 27. Container logs, and so the
clone output in the build log and the app logs, no longer carry Docker's
stream frame headers, and the commit is now read from the clone output (#251).
- 2026-10-01: An app's first deploy no longer fails when a volume's host path
does not exist yet: upaas asks Docker to create a missing host path when the
app's container starts and to leave an existing one alone, so nobody has to
create it on the host first. The README has a Volume mounts section saying so
(#235).
- 2026-09-29: The app page is 50% wider on large screens (84rem instead of
56rem), its build log and container log boxes are twice as tall, the build log
sits between the webhook URL and the environment variables, and the container
log sits above the deploy key (#246).
- 2026-09-29: A failed build now fails the deploy with the build's own error
instead of a later "failed to inspect image", and the deployment log shows the
end of the build output before that error. upaas refuses to build on a Docker
Engine older than 18.09, which cannot build with BuildKit. The README's
Compose section says to update with `docker compose up -d --build` and names
the Docker Engine versions builds need (#234).
- 2026-09-29: An image built from the `Dockerfile` now shows the commit it was
built from (the `git describe` output) in the footer and `/health` instead of
`dev`: `.dockerignore` no longer leaves out `.git`, nor any tracked file,
which git would count as deleted and mark `-dirty`. upaas now also logs its
version at startup; the logger's `Identify()` was never called (#236).
- 2026-09-29: The app page shows the app's branch as a label in its title, next
to the status badge, instead of after the repository under it (#240).
- 2026-09-29: An app's deployments page now lists only its 10 most recent
deployments, newest first, instead of 50 (#238).
- 2026-09-29: `docker-compose.yml` now sets `UPAAS_PORT` to 8080 as well as
`PORT`, since upaas reads `UPAAS_PORT` first and a `UPAAS_PORT` in `.env` made
it listen away from the port mapping and healthcheck; the README's Compose
section names both (#230).
- 2026-09-29: The README Configuration table now lists every setting upaas
reads, adding `UPAAS_MAINTENANCE_MODE`, `UPAAS_SESSION_SECRET` and
`UPAAS_CORS_ORIGINS`, and gives the real default and effect of each:
`UPAAS_PORT` wins over `PORT`, `UPAAS_HOST_DATA_DIR` falls back to
`UPAAS_DATA_DIR`, `UPAAS_DEBUG` drops the session cookie's `Secure` flag, and
`UPAAS_SENTRY_DSN` is not used (#229).
- 2026-09-28: The README Configuration table now names `UPAAS_DEBUG`,
`UPAAS_SENTRY_DSN`, `UPAAS_METRICS_USERNAME` and `UPAAS_METRICS_PASSWORD`, the
names upaas actually reads (the unprefixed names it listed were ignored), and
the `UPAAS_HOST_DATA_DIR` row refers to `UPAAS_DATA_DIR` (#224).
- 2026-09-28: Added `docker-compose.yml` for deploying upaas: settings from
`.env`, the port published on `127.0.0.1` only for a TLS proxy in front, and a
healthcheck against `/health`; the README's plain-HTTP Compose example is
replaced by a short deploy section. upaas now refuses to start when
`UPAAS_HOST_DATA_DIR` is set to a relative path (#223).
- 2026-09-23: Apps are now built with BuildKit, so the stages of a multi-stage
build stay in Docker's size-limited build cache instead of piling up as
untagged images; build progress is still written to the deployment log as
plain text (#220).
- 2026-09-23: After a successful deploy, upaas removes the app's images other
than the running one and the one rollback would use, together with the
untagged images they were built on (#216).
- 2026-09-23: The git clone container is now removed together with its anonymous
volume (the `alpine/git` image declares one at `/git`), so a deploy no longer
leaves a Docker volume behind (#215).
- 2026-09-23: Deployment log files are now stored under `logs/<appname>/`
instead of `logs/<hostname>/<appname>/`, so downloads keep working after the
upaas container is recreated; logs written under an old hostname directory are
still found (#214).
- 2026-09-23: Fixed the flaky `t.TempDir` cleanup race in `internal/handlers`
(the one fixed in `internal/service/webhook` by #198):
`TestHandleWebhookProcessesValidWebhook` now waits with the webhook service's
`WaitForDeployments` instead of sleeping (#211).
- 2026-09-22: Vendored the canonical prettier/format toolchain from the
`sneak/prompts` scaffold: added `.prettierrc` (tabWidth 4, proseWrap always),
pinned `package.json` + `yarn.lock` (prettier 3.8.1), taught
`script/bootstrap` to install a pinned node/yarn via a hash-verified nvm
archive, and switched `script/fmt` to the pinned prettier reading
`.prettierrc` (no inline flags) over `static/js/*.js` and `**/*.md`. Reflowed
all markdown to house style; `alpine.min.js` stays byte-identical (#203).
- 2026-09-22: Fixed the flaky `t.TempDir` cleanup race in
`internal/service/webhook` by tracking the async deployment goroutine in a
`sync.WaitGroup` and exposing `WaitForDeployments`; tests now synchronize on
completion instead of sleeping (#198).
- 2026-09-22: Linting now runs only in Docker. Added `Dockerfile.lint` (pinned
golangci-lint v2.12.2, cache-busted via a `GATE_RUN` build arg so the linter
always executes), reduced `script/lint` to building it, dropped the
golangci-lint install from `script/bootstrap`, and switched the `Dockerfile`
lint stage to invoke `golangci-lint` directly instead of `make lint` to avoid
docker-in-docker (#188).
- 2026-09-22: Added `.prettierignore` so `make fmt` no longer rewrites the
vendored `static/js/alpine.min.js` bundle (#185).
- 2026-09-22: Fixed the gosec G703 path-traversal finding in the deploy log
download handler by verifying the resolved path stays within the deploy log
directory before serving, returning 404 on escape (#177).
- 2026-09-22: `script/bootstrap` now installs a pinned `goimports`
(`golang.org/x/tools` v0.49.0) into `/usr/local/bin`, so `make fmt` succeeds
on a fresh machine after `make bootstrap` (#184).
- 2026-09-09: Fixed four deployability blockers found by QA: CSRF origin check
over plain HTTP (`UPAAS_PLAINTEXT_HTTP`, #189), pulling the git image when
absent (#190), the env-var editor CSRF token lookup (#191), and the
port-mapping delete form's CSRF field (#192).
- 2026-08-07: Updated golangci-lint to v2.12.2 (canonical `.golangci.yml`,
`Dockerfile` lint stage pin, `script/bootstrap` release-archive pins) and
fixed all resulting lint findings (noctx, gosec, goconst, lll, dupl,
nolintlint); `make check` green.
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
shims, README Entrypoints section
- 2026-03-11: Monolithic env var editing with bulk save (#158).
- 2026-03-10: Webhook event history UI page (#164); added missing Makefile
docker and hooks targets plus test timeout (#159); notification settings
passed from create form (#160).
- 2026-03-03: REPO_POLICIES compliance file set added (#155).
- 2026-03-01: Module path changed to sneak.berlin/go/upaas (#143); Dockerfile
split into lint and build stages with forced lint execution (#152, #154).
- 2026-02-26: 1.0.0 tagged; dashboard CSRFField crash fixed (#146).
- 1.0 audit bug fixes (#120-#125): deferred rollback on commit error, deployment
log size cap, error path rendering, docker-compose bind mount, domain type
refactor.
- CI simplified to docker build only (#130).
- 2025-12-29 onward: core PaaS built out: deploys with real-time build log
streaming, container start/stop/restart and logs, TCP/UDP port mapping,
Alpine.js UI, Slack notifications, ULID app IDs, session handling.
# Future Steps
- Resume feature work only after main is green.
+4 -2
View File
@@ -25,13 +25,15 @@ import (
// Build-time variables injected by linker flags (-ldflags).
// These must be exported package-level variables for the build system.
var (
Appname = "upaas" //nolint:gochecknoglobals // build-time variable
Version string //nolint:gochecknoglobals // build-time variable
Appname = "upaas" //nolint:gochecknoglobals // build-time variable
Version string //nolint:gochecknoglobals // build-time variable
Buildarch string //nolint:gochecknoglobals // build-time variable
)
func main() {
globals.SetAppname(Appname)
globals.SetVersion(Version)
globals.SetBuildarch(Buildarch)
fx.New(
fx.Provide(
+2 -2
View File
@@ -4,7 +4,6 @@ go 1.25
require (
github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8
github.com/distribution/reference v0.6.0
github.com/docker/docker v27.3.1+incompatible
github.com/docker/go-connections v0.6.0
github.com/go-chi/chi/v5 v5.2.3
@@ -15,7 +14,6 @@ require (
github.com/joho/godotenv v1.5.1
github.com/mattn/go-sqlite3 v1.14.32
github.com/moby/buildkit v0.16.0
github.com/moby/patternmatcher v0.6.0
github.com/oklog/ulid/v2 v2.1.1
github.com/prometheus/client_golang v1.23.2
github.com/spf13/viper v1.21.0
@@ -41,6 +39,7 @@ require (
github.com/containerd/ttrpc v1.2.5 // indirect
github.com/containerd/typeurl/v2 v2.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
@@ -61,6 +60,7 @@ require (
github.com/klauspost/compress v1.18.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/signal v0.7.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
-114
View File
@@ -1,114 +0,0 @@
package docker
import (
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"github.com/docker/docker/pkg/archive"
"github.com/moby/patternmatcher"
"github.com/moby/patternmatcher/ignorefile"
)
// defaultDockerfileName is the Dockerfile Docker builds when none is named.
const defaultDockerfileName = "Dockerfile"
// defaultDockerignoreName is the ignore file at the root of a build context,
// read when the Dockerfile has no ignore file of its own.
const defaultDockerignoreName = ".dockerignore"
// tarBuildContext returns a tar of the build context in contextDir that leaves
// out the files the app's ignore file names, as docker build does; Docker does
// not apply the ignore file to a build context sent as a tar. dockerfile is
// the path of the Dockerfile inside contextDir.
func tarBuildContext(contextDir, dockerfile string) (io.ReadCloser, error) {
excludes, err := readDockerignore(contextDir, dockerfile)
if err != nil {
return nil, err
}
return archive.TarWithOptions(contextDir, &archive.TarOptions{
ExcludePatterns: excludes,
})
}
// readDockerignore returns the patterns of the files to leave out of the
// build context in contextDir, read as docker build reads them for the
// Dockerfile at dockerfile: from <dockerfile>.dockerignore next to the
// Dockerfile if there is one, otherwise from .dockerignore at the root of the
// context. Without either file there are no patterns.
func readDockerignore(contextDir, dockerfile string) ([]string, error) {
// Docker reads the Dockerfile path as a path inside the build context:
// cleaned, with a leading / and any .. that would lead out of the context
// dropped, so ./Dockerfile and /Dockerfile both name the Dockerfile at the
// root.
dockerfile = strings.TrimPrefix(filepath.Join("/", dockerfile), "/")
if dockerfile == "" {
dockerfile = defaultDockerfileName
}
// Reading through os.Root keeps the read inside the build context, even
// when the ignore file is a symlink.
root, err := os.OpenRoot(contextDir)
if err != nil {
return nil, err
}
defer func() { _ = root.Close() }()
// When a Dockerfile named Dockerfile is missing, Docker builds a
// lowercase dockerfile in the same directory instead, and docker build
// then reads dockerfile.dockerignore as its ignore file.
if filepath.Base(dockerfile) == defaultDockerfileName {
lowercase := filepath.Join(filepath.Dir(dockerfile), "dockerfile")
_, dockerfileErr := root.Lstat(dockerfile)
_, lowercaseErr := root.Lstat(lowercase)
if errors.Is(dockerfileErr, fs.ErrNotExist) && lowercaseErr == nil {
dockerfile = lowercase
}
}
name := dockerfile + defaultDockerignoreName
file, err := root.Open(name)
if errors.Is(err, fs.ErrNotExist) {
name = defaultDockerignoreName
file, err = root.Open(name)
}
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to read %s: %w", name, err)
}
defer func() { _ = file.Close() }()
excludes, err := ignorefile.ReadAll(file)
if err != nil {
return nil, fmt.Errorf("failed to read %s: %w", name, err)
}
// Like the docker command line, never leave out .dockerignore or the
// Dockerfile: Docker reads the Dockerfile from the context.
for _, keep := range []string{defaultDockerignoreName, filepath.ToSlash(dockerfile)} {
excluded, err := patternmatcher.MatchesOrParentMatches(keep, excludes)
if err != nil {
return nil, fmt.Errorf("invalid pattern in %s: %w", name, err)
}
if excluded {
excludes = append(excludes, "!"+keep)
}
}
return excludes, nil
}
-326
View File
@@ -1,326 +0,0 @@
package docker //nolint:testpackage // tests the unexported performBuild
import (
"archive/tar"
"errors"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/docker/docker/client"
)
// File names used in more than one test build context, as constants to
// satisfy the goconst linter.
const (
testMainGo = "main.go"
testSecretFile = "secret.txt"
testDeployDockerfile = "deploy/Dockerfile"
testLowercaseDockerfile = "dockerfile"
)
// TestPerformBuildFollowsDockerignore runs builds against a fake Docker API
// and checks which files the build context sent to it holds.
func TestPerformBuildFollowsDockerignore(t *testing.T) {
t.Parallel()
tests := []struct {
name string
dockerfile string
files map[string]string // path in the context: contents
want []string // files sent in the build context
}{
{
name: "no ignore file",
dockerfile: defaultDockerfileName,
files: map[string]string{defaultDockerfileName: "", testMainGo: ""},
want: []string{defaultDockerfileName, testMainGo},
},
{
name: "excludes and re-includes",
dockerfile: defaultDockerfileName,
files: map[string]string{
defaultDockerignoreName: "secret.txt\n*.md\n!README.md\n",
defaultDockerfileName: "",
"NOTES.md": "",
"README.md": "",
testMainGo: "",
testSecretFile: "",
},
want: []string{
defaultDockerignoreName, defaultDockerfileName, "README.md", testMainGo,
},
},
{
name: "keeps the Dockerfile and .dockerignore",
dockerfile: defaultDockerfileName,
files: map[string]string{
defaultDockerignoreName: "Dockerfile\n.dockerignore\nsecret.txt\n",
defaultDockerfileName: "",
testMainGo: "",
testSecretFile: "",
},
want: []string{defaultDockerignoreName, defaultDockerfileName, testMainGo},
},
{
name: "an ignore file next to the Dockerfile wins over .dockerignore",
dockerfile: testDeployDockerfile,
files: map[string]string{
defaultDockerignoreName: "main.go\n",
testDeployDockerfile: "",
"deploy/Dockerfile.dockerignore": "secret.txt\n",
testMainGo: "",
testSecretFile: "",
},
want: []string{
defaultDockerignoreName,
testDeployDockerfile,
"deploy/Dockerfile.dockerignore",
testMainGo,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
contextDir := t.TempDir()
writeFiles(t, contextDir, tt.files)
got, err := buildContextFiles(t, contextDir, tt.dockerfile)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(got, tt.want) {
t.Errorf("build context holds %q, want %q", got, tt.want)
}
})
}
}
// TestPerformBuildKeepsLowercaseDockerfile checks that when the Dockerfile
// named Dockerfile is missing, the lowercase dockerfile Docker builds instead
// stays in the build context, and its own ignore file is read.
func TestPerformBuildKeepsLowercaseDockerfile(t *testing.T) {
t.Parallel()
tests := []struct {
name string
files map[string]string // path in the context: contents
want []string // files sent in the build context
}{
{
name: "kept when the ignore file names it",
files: map[string]string{
defaultDockerignoreName: "*\n",
testLowercaseDockerfile: "",
},
want: []string{defaultDockerignoreName, testLowercaseDockerfile},
},
{
name: "its own ignore file wins over .dockerignore",
files: map[string]string{
defaultDockerignoreName: "main.go\n",
testLowercaseDockerfile: "",
"dockerfile.dockerignore": "secret.txt\n",
testMainGo: "",
testSecretFile: "",
},
want: []string{
defaultDockerignoreName,
testLowercaseDockerfile,
"dockerfile.dockerignore",
testMainGo,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
contextDir := t.TempDir()
writeFiles(t, contextDir, tt.files)
got, err := buildContextFiles(t, contextDir, defaultDockerfileName)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(got, tt.want) {
t.Errorf("build context holds %q, want %q", got, tt.want)
}
})
}
}
// TestPerformBuildReadsDockerfilePathInsideContext checks that ./Dockerfile
// and /Dockerfile name the Dockerfile at the root of the context, as Docker
// reads them, so an ignore file that names the Dockerfile does not leave it
// out.
func TestPerformBuildReadsDockerfilePathInsideContext(t *testing.T) {
t.Parallel()
for _, dockerfile := range []string{"./Dockerfile", "/Dockerfile"} {
t.Run(dockerfile, func(t *testing.T) {
t.Parallel()
contextDir := t.TempDir()
writeFiles(t, contextDir, map[string]string{
defaultDockerignoreName: "Dockerfile\nsecret.txt\n",
defaultDockerfileName: "",
testMainGo: "",
testSecretFile: "",
})
got, err := buildContextFiles(t, contextDir, dockerfile)
if err != nil {
t.Fatal(err)
}
want := []string{defaultDockerignoreName, defaultDockerfileName, testMainGo}
if !slices.Equal(got, want) {
t.Errorf("build context holds %q, want %q", got, want)
}
})
}
}
// TestPerformBuildFailsOnMalformedDockerignore checks that a pattern the
// docker command line would reject fails the build.
func TestPerformBuildFailsOnMalformedDockerignore(t *testing.T) {
t.Parallel()
contextDir := t.TempDir()
writeFiles(t, contextDir, map[string]string{defaultDockerignoreName: "[\n"})
_, err := buildContextFiles(t, contextDir, defaultDockerfileName)
want := "failed to create build context: " +
"invalid pattern in .dockerignore: syntax error in pattern"
if err == nil || err.Error() != want {
t.Errorf("got error %v, want %q", err, want)
}
}
// TestPerformBuildFailsOnUnreadableDockerignore checks that an ignore file
// that cannot be read, here because it is a directory, fails the build.
func TestPerformBuildFailsOnUnreadableDockerignore(t *testing.T) {
t.Parallel()
contextDir := t.TempDir()
err := os.Mkdir(filepath.Join(contextDir, defaultDockerignoreName), 0o750)
if err != nil {
t.Fatal(err)
}
_, err = buildContextFiles(t, contextDir, defaultDockerfileName)
want := "failed to create build context: failed to read .dockerignore: "
if err == nil || !strings.HasPrefix(err.Error(), want) {
t.Errorf("got error %v, want one starting %q", err, want)
}
}
// writeFiles writes files, a map of paths inside dir to their contents,
// creating the directories they are in.
func writeFiles(t *testing.T, dir string, files map[string]string) {
t.Helper()
for name, contents := range files {
path := filepath.Join(dir, name)
err := os.MkdirAll(filepath.Dir(path), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.WriteFile(path, []byte(contents), 0o600)
if err != nil {
t.Fatal(err)
}
}
}
// buildContextFiles runs a build of contextDir against a fake Docker API and
// returns the names of the files in the build context sent to it, sorted.
func buildContextFiles(t *testing.T, contextDir, dockerfile string) ([]string, error) {
t.Helper()
sent := make(chan []string, 1)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/version"):
_, _ = w.Write([]byte(`{"Version":"27.3.1","ApiVersion":"1.47"}`))
case strings.HasSuffix(r.URL.Path, "/session"):
serveSession(t, w, r, make(chan string, 1))
case strings.HasSuffix(r.URL.Path, "/build"):
sent <- tarFileNames(t, r.Body)
default:
t.Errorf("unexpected request to %s", r.URL.Path)
}
},
))
t.Cleanup(srv.Close)
dockerAPI, err := client.NewClientWithOpts(
client.WithHost("tcp://" + srv.Listener.Addr().String()),
)
if err != nil {
t.Fatal(err)
}
c := &Client{docker: dockerAPI, log: slog.Default()}
_, err = c.performBuild(t.Context(), BuildImageOptions{
ContextDir: contextDir,
DockerfilePath: dockerfile,
})
if err != nil {
return nil, err
}
return <-sent, nil
}
// tarFileNames returns the names of the regular files in the tar read from r,
// sorted.
func tarFileNames(t *testing.T, r io.Reader) []string {
t.Helper()
var names []string
reader := tar.NewReader(r)
for {
header, err := reader.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
t.Errorf("reading the build context: %v", err)
return nil
}
if header.Typeflag == tar.TypeReg {
names = append(names, header.Name)
}
}
slices.Sort(names)
return names
}
+20 -61
View File
@@ -25,6 +25,7 @@ import (
"github.com/docker/docker/api/types/network"
"github.com/docker/docker/api/types/versions"
"github.com/docker/docker/client"
"github.com/docker/docker/pkg/archive"
"github.com/docker/docker/pkg/jsonmessage"
"github.com/docker/docker/pkg/stdcopy"
"github.com/docker/go-connections/nat"
@@ -504,7 +505,6 @@ type cloneConfig struct {
type CloneResult struct {
Output string // Combined stdout/stderr from git clone
CommitSHA string // The HEAD commit SHA after clone/checkout
ShortSHA string // git's short form of CommitSHA (git rev-parse --short)
}
// CloneRepo clones a git repository using SSH and optionally checks out a
@@ -567,7 +567,7 @@ func (c *Client) RemoveImage(ctx context.Context, imageID ImageID) error {
}
// ListImageTags returns the tags in the given repository, such as
// "upaas-myapp:1a2b3c4" in "upaas-myapp", each with the ID of its image.
// "upaas-myapp:12" in "upaas-myapp", each with the ID of its image.
// Tags the same image has in other repositories are left out.
func (c *Client) ListImageTags(
ctx context.Context,
@@ -597,44 +597,19 @@ func (c *Client) ListImageTags(
return tags, nil
}
// ListUntaggedImages returns the IDs of the images that have no tag, such
// as one whose tag a later build gave to the image it built.
func (c *Client) ListUntaggedImages(ctx context.Context) ([]ImageID, error) {
if c.docker == nil {
return nil, ErrNotConnected
}
images, err := c.docker.ImageList(ctx, image.ListOptions{
Filters: filters.NewArgs(filters.Arg("dangling", "true")),
})
if err != nil {
return nil, fmt.Errorf("failed to list untagged images: %w", err)
}
imageIDs := make([]ImageID, 0, len(images))
for _, img := range images {
imageIDs = append(imageIDs, ImageID(img.ID))
}
return imageIDs, nil
}
// RemoveImageTag removes the tag name, such as "upaas-myapp:1a2b3c4",
// without force. Docker then deletes the image, and the untagged images it
// was built on, only if no other tag and no container still uses it. If name
// is instead the ID of an untagged image, it removes that image unless a
// container uses it.
func (c *Client) RemoveImageTag(ctx context.Context, name string) error {
// RemoveImageTag removes a tag such as "upaas-myapp:12", without force.
// Docker then deletes the image, and the untagged images it was built on,
// only if no other tag and no container still uses it.
func (c *Client) RemoveImageTag(ctx context.Context, tag string) error {
if c.docker == nil {
return ErrNotConnected
}
_, err := c.docker.ImageRemove(ctx, name, image.RemoveOptions{
_, err := c.docker.ImageRemove(ctx, tag, image.RemoveOptions{
PruneChildren: true,
})
if err != nil && !client.IsErrNotFound(err) {
return fmt.Errorf("failed to remove image %s: %w", name, err)
return fmt.Errorf("failed to remove image tag %s: %w", tag, err)
}
return nil
@@ -659,7 +634,7 @@ func (c *Client) performBuild(
}
// Create tar archive of build context
tarArchive, err := tarBuildContext(opts.ContextDir, opts.DockerfilePath)
tarArchive, err := archive.TarWithOptions(opts.ContextDir, &archive.TarOptions{})
if err != nil {
return "", fmt.Errorf("failed to create build context: %w", err)
}
@@ -930,13 +905,11 @@ func (c *Client) createGitContainer(
// Clone without depth limit so we can checkout any commit, then checkout specific SHA
script = `git clone --branch "$CLONE_BRANCH" "$CLONE_URL" /repo` +
` && cd /repo && git checkout "$CLONE_SHA"` +
` && echo COMMIT:$(git rev-parse HEAD)` +
` && echo SHORT_SHA:$(git rev-parse --short HEAD)`
` && echo COMMIT:$(git rev-parse HEAD)`
} else {
// Shallow clone of branch HEAD, then output commit SHA
script = `git clone --depth 1 --branch "$CLONE_BRANCH" "$CLONE_URL" /repo` +
` && cd /repo && echo COMMIT:$(git rev-parse HEAD)` +
` && echo SHORT_SHA:$(git rev-parse --short HEAD)`
` && cd /repo && echo COMMIT:$(git rev-parse HEAD)`
}
env := []string{
@@ -1014,37 +987,23 @@ func (c *Client) runGitClone(
)
}
// Parse the commit from the "COMMIT:" and "SHORT_SHA:" lines.
result := &CloneResult{
Output: logs,
CommitSHA: parseCommitSHA(logs, commitMarker),
ShortSHA: parseCommitSHA(logs, shortSHAMarker),
}
// Parse commit SHA from output (looks for "COMMIT:<sha>" line)
commitSHA := parseCommitSHA(logs)
// The short hash names the image the deploy builds.
if result.ShortSHA == "" {
return nil, fmt.Errorf("%w: no short commit hash in its output: %s",
ErrGitCloneFailed, logs)
}
return result, nil
return &CloneResult{Output: logs, CommitSHA: commitSHA}, nil
}
}
// Prefixes of the lines in the clone output that carry the commit checked
// out, in full and in git's short form.
const (
commitMarker = "COMMIT:"
shortSHAMarker = "SHORT_SHA:"
)
// commitMarker is the prefix used to identify commit SHA in clone output.
const commitMarker = "COMMIT:"
// parseCommitSHA extracts a commit SHA from git clone output.
// It looks for a line starting with marker and returns the SHA after it.
func parseCommitSHA(output, marker string) string {
// parseCommitSHA extracts the commit SHA from git clone output.
// It looks for a line starting with "COMMIT:" and returns the SHA after it.
func parseCommitSHA(output string) string {
for line := range strings.SplitSeq(output, "\n") {
line = strings.TrimSpace(line)
sha, found := strings.CutPrefix(line, marker)
sha, found := strings.CutPrefix(line, commitMarker)
if found {
return strings.TrimSpace(sha)
}
+34 -152
View File
@@ -16,7 +16,6 @@ import (
"testing"
"time"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/client"
"github.com/docker/docker/pkg/stdcopy"
controlapi "github.com/moby/buildkit/api/services/control"
@@ -206,8 +205,6 @@ func TestPerformCloneRemovesContainerVolumes(t *testing.T) {
<-r.Context().Done()
case strings.HasSuffix(r.URL.Path, "/wait"):
_, _ = fmt.Fprintf(w, `{"StatusCode":%d}`, tt.exitCode)
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w)
default:
_, _ = w.Write([]byte(`{}`))
}
@@ -224,7 +221,18 @@ func TestPerformCloneRemovesContainerVolumes(t *testing.T) {
c := &Client{docker: dockerAPI, log: slog.Default()}
_, _ = c.performClone(ctx, testCloneConfig(t))
dir := t.TempDir()
cfg := &cloneConfig{
repoURL: "git@example.com:repo.git",
branch: mainBranch,
sshPrivateKey: "fake-key",
containerDir: filepath.Join(dir, "repo"),
hostDir: filepath.Join(dir, "repo"),
keyFile: filepath.Join(dir, "deploy_key"),
hostKeyFile: filepath.Join(dir, "deploy_key"),
}
_, _ = c.performClone(ctx, cfg)
select {
case query := <-removeQuery:
@@ -238,38 +246,6 @@ func TestPerformCloneRemovesContainerVolumes(t *testing.T) {
}
}
// testCloneConfig returns the settings of a clone in these tests, with its
// files in a temporary directory.
func testCloneConfig(t *testing.T) *cloneConfig {
t.Helper()
dir := t.TempDir()
return &cloneConfig{
repoURL: "git@example.com:repo.git",
branch: mainBranch,
sshPrivateKey: "fake-key",
containerDir: filepath.Join(dir, "repo"),
hostDir: filepath.Join(dir, "repo"),
keyFile: filepath.Join(dir, "deploy_key"),
hostKeyFile: filepath.Join(dir, "deploy_key"),
}
}
// cloneCommit is the commit the fake clones in these tests check out.
const cloneCommit = "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b"
// writeCloneOutput writes the output of a git clone of cloneCommit as
// Docker sends a container's log: each line after a header.
func writeCloneOutput(w io.Writer) {
stdout := stdcopy.NewStdWriter(w, stdcopy.Stdout)
stderr := stdcopy.NewStdWriter(w, stdcopy.Stderr)
_, _ = stderr.Write([]byte("Cloning into '/repo'...\n"))
_, _ = stdout.Write([]byte("COMMIT:" + cloneCommit + "\n"))
_, _ = stdout.Write([]byte("SHORT_SHA:1a2b3c4\n"))
}
// TestPerformBuildUsesBuildKit runs a build against a fake Docker API and
// checks that it asks for BuildKit and that BuildKit's progress reaches the
// build log as plain text.
@@ -525,11 +501,12 @@ func serveSession(
// TestPerformCloneReadsFramedLogs runs a clone against a fake Docker API that
// sends the clone container's output in frames, as Docker does for a
// container without a terminal, and checks that the output comes back as
// plain text and that the commit, in full and in git's short form, is read
// from it.
// plain text and that the commit is read from it.
func TestPerformCloneReadsFramedLogs(t *testing.T) {
t.Parallel()
const commit = "1647b43aa6b211686719313bc6372c3693c54ca9"
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
@@ -540,7 +517,10 @@ func TestPerformCloneReadsFramedLogs(t *testing.T) {
case strings.HasSuffix(r.URL.Path, "/wait"):
_, _ = w.Write([]byte(`{"StatusCode":0}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w)
_, _ = stdcopy.NewStdWriter(w, stdcopy.Stderr).
Write([]byte("Cloning into '/repo'...\n"))
_, _ = stdcopy.NewStdWriter(w, stdcopy.Stdout).
Write([]byte("COMMIT:" + commit + "\n"))
default:
_, _ = w.Write([]byte(`{}`))
}
@@ -557,126 +537,28 @@ func TestPerformCloneReadsFramedLogs(t *testing.T) {
c := &Client{docker: dockerAPI, log: slog.Default()}
result, err := c.performClone(t.Context(), testCloneConfig(t))
dir := t.TempDir()
cfg := &cloneConfig{
repoURL: "git@example.com:repo.git",
branch: mainBranch,
sshPrivateKey: "fake-key",
containerDir: filepath.Join(dir, "repo"),
hostDir: filepath.Join(dir, "repo"),
keyFile: filepath.Join(dir, "deploy_key"),
hostKeyFile: filepath.Join(dir, "deploy_key"),
}
result, err := c.performClone(t.Context(), cfg)
if err != nil {
t.Fatal(err)
}
want := "Cloning into '/repo'...\nCOMMIT:" + cloneCommit + "\nSHORT_SHA:1a2b3c4\n"
want := "Cloning into '/repo'...\nCOMMIT:" + commit + "\n"
if result.Output != want {
t.Errorf("got clone output %q, want %q", result.Output, want)
}
if result.CommitSHA != cloneCommit || result.ShortSHA != "1a2b3c4" {
t.Errorf("got commit %q, short %q", result.CommitSHA, result.ShortSHA)
}
}
// TestPerformCloneFailsWithoutShortSHA runs a clone against a fake Docker API
// whose clone succeeds but prints no "SHORT_SHA:" line, and checks that the
// clone fails, since the short hash names the image the deploy builds.
func TestPerformCloneFailsWithoutShortSHA(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case strings.HasSuffix(r.URL.Path, "/containers/create"):
_, _ = w.Write([]byte(`{"Id":"gitcontainer"}`))
case strings.HasSuffix(r.URL.Path, "/wait"):
_, _ = w.Write([]byte(`{"StatusCode":0}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
_, _ = stdcopy.NewStdWriter(w, stdcopy.Stdout).
Write([]byte("COMMIT:" + cloneCommit + "\n"))
default:
_, _ = w.Write([]byte(`{}`))
}
},
))
t.Cleanup(srv.Close)
dockerAPI, err := client.NewClientWithOpts(
client.WithHost("tcp://" + srv.Listener.Addr().String()),
)
if err != nil {
t.Fatal(err)
}
c := &Client{docker: dockerAPI, log: slog.Default()}
_, err = c.performClone(t.Context(), testCloneConfig(t))
if !errors.Is(err, ErrGitCloneFailed) {
t.Errorf("got error %v, want %v", err, ErrGitCloneFailed)
}
}
// TestPerformCloneAsksGitForShortSHA runs a clone of a branch's last commit
// and a clone of a given commit against a fake Docker API, and checks that
// the command each clone container is created with prints git's own short
// form of the commit checked out.
func TestPerformCloneAsksGitForShortSHA(t *testing.T) {
t.Parallel()
tests := []struct {
name string
commitSHA string
}{
{name: "branch", commitSHA: ""},
{name: "commit", commitSHA: cloneCommit},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
created := make(chan container.Config, 1)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case strings.HasSuffix(r.URL.Path, "/containers/create"):
var cfg container.Config
_ = json.NewDecoder(r.Body).Decode(&cfg)
created <- cfg
_, _ = w.Write([]byte(`{"Id":"gitcontainer"}`))
case strings.HasSuffix(r.URL.Path, "/wait"):
_, _ = w.Write([]byte(`{"StatusCode":0}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w)
default:
_, _ = w.Write([]byte(`{}`))
}
},
))
t.Cleanup(srv.Close)
dockerAPI, err := client.NewClientWithOpts(
client.WithHost("tcp://" + srv.Listener.Addr().String()),
)
if err != nil {
t.Fatal(err)
}
c := &Client{docker: dockerAPI, log: slog.Default()}
cfg := testCloneConfig(t)
cfg.commitSHA = tt.commitSHA
_, err = c.performClone(t.Context(), cfg)
if err != nil {
t.Fatal(err)
}
cmd := strings.Join((<-created).Cmd, " ")
if !strings.Contains(cmd, "echo SHORT_SHA:$(git rev-parse --short HEAD)") {
t.Errorf("clone command %q does not print git's short hash", cmd)
}
})
if result.CommitSHA != commit {
t.Errorf("got commit %q, want %q", result.CommitSHA, commit)
}
}
+18 -7
View File
@@ -12,15 +12,17 @@ import (
//
//nolint:gochecknoglobals // Required for ldflags injection at build time
var (
mu sync.RWMutex
appname string
version string
mu sync.RWMutex
appname string
version string
buildarch string
)
// Globals holds build-time variables for dependency injection.
type Globals struct {
Appname string
Version string
Appname string
Version string
Buildarch string
}
// New creates a new Globals instance from package-level variables.
@@ -29,8 +31,9 @@ func New(_ fx.Lifecycle) (*Globals, error) {
defer mu.RUnlock()
return &Globals{
Appname: appname,
Version: version,
Appname: appname,
Version: version,
Buildarch: buildarch,
}, nil
}
@@ -49,3 +52,11 @@ func SetVersion(ver string) {
version = ver
}
// SetBuildarch sets the build architecture (used for testing and main init).
func SetBuildarch(arch string) {
mu.Lock()
defer mu.Unlock()
buildarch = arch
}
+2
View File
@@ -21,6 +21,7 @@ import (
"sneak.berlin/go/upaas/internal/database"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/app"
"sneak.berlin/go/upaas/internal/service/deploy"
"sneak.berlin/go/upaas/templates"
)
@@ -194,6 +195,7 @@ func (h *Handlers) HandleAppDetail() http.HandlerFunc {
"EnvVars": envVars,
"Labels": labels,
"Volumes": volumes,
"NoVolumesWarning": deploy.NoVolumesWarning,
"Ports": ports,
"Deployments": deployments,
"LatestDeployment": latestDeployment,
+7 -13
View File
@@ -13,15 +13,12 @@ const (
appNameMaxLength = 63
)
// validAppNameRe matches runs of lowercase letters and digits joined by
// single dots or by hyphens, such as "my-app" or "sneak.berlin". Docker
// accepts every name it allows as the app's image name, upaas-<name>; a
// dot needs a letter or digit on both sides because Docker requires it.
// It also keeps the name from being "." or ".." or starting or ending
// with a dot, so it is safe as a directory and file name. The pattern
// attribute of the name field on the new and edit app forms is the same.
var validAppNameRe = regexp.MustCompile(`^[a-z0-9]+((\.|-+)[a-z0-9]+)*$`)
// validAppNameRe matches names containing only lowercase alphanumeric characters and
// hyphens, starting and ending with an alphanumeric character.
var validAppNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*[a-z0-9]$`)
// validateAppName checks that the given app name is safe for use in Docker
// container names, image tags, and file system paths.
var (
errAppNameLength = errors.New(
"app name must be between " +
@@ -29,14 +26,11 @@ var (
strconv.Itoa(appNameMaxLength) + " characters",
)
errAppNamePattern = errors.New(
"app name must contain only lowercase letters, numbers, hyphens, " +
"and dots, must start and end with a letter or number, " +
"and must have a letter or number on both sides of each dot",
"app name must contain only lowercase letters, numbers, " +
"and hyphens, and must start and end with a letter or number",
)
)
// validateAppName checks that the given app name is safe for use in Docker
// container names, image tags, and file system paths.
func validateAppName(name string) error {
if len(name) < appNameMinLength || len(name) > appNameMaxLength {
return errAppNameLength
+1 -58
View File
@@ -1,16 +1,7 @@
package handlers //nolint:testpackage // testing unexported validateAppName
import (
"bytes"
"strconv"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/templates"
)
func TestValidateAppName(t *testing.T) {
@@ -27,10 +18,6 @@ func TestValidateAppName(t *testing.T) {
{"valid two chars", "ab", false},
{"valid complex", "my-cool-app-v2", false},
{"valid all numbers", "123", false},
{"valid double hyphen", "my--app", false},
{"valid domain", "sneak.berlin", false},
{"valid two dots", "www.sneak.berlin", false},
{"valid dot and hyphen", "my-app.example.com", false},
{"empty", "", true},
{"single char", "a", true},
{"too long", "a" + string(make([]byte, 63)), true},
@@ -49,12 +36,7 @@ func TestValidateAppName(t *testing.T) {
{"starts with hyphen", "-myapp", true},
{"ends with hyphen", "myapp-", true},
{"underscore", "my_app", true},
{"two dots in a row", "a..b", true},
{"starts with dot", ".a", true},
{"ends with dot", "a.", true},
{"only dots", "..", true},
{"hyphen before dot", "a-.b", true},
{"hyphen after dot", "a.-b", true},
{"dot", "my.app", true},
{"slash", "my/app", true},
{"path traversal", "../etc/passwd", true},
{"special chars", "app@name!", true},
@@ -72,42 +54,3 @@ func TestValidateAppName(t *testing.T) {
})
}
}
func TestValidateAppNameErrorMentionsDots(t *testing.T) {
t.Parallel()
err := validateAppName("a..b")
require.ErrorIs(t, err, errAppNamePattern)
assert.Contains(t, err.Error(), "dots")
}
// TestAppFormsCheckAppNameLikeServer checks that the name field on the new
// and edit app forms has the server's pattern and length limits, and that
// its hint mentions dots.
func TestAppFormsCheckAppNameLikeServer(t *testing.T) {
t.Parallel()
pattern := strings.TrimSuffix(strings.TrimPrefix(validAppNameRe.String(), "^"), "$")
pages := map[string]map[string]any{
"app_new.html": {},
"app_edit.html": {dataKeyApp: &models.App{}},
}
for page, data := range pages {
var out bytes.Buffer
require.NoError(t, templates.GetParsed().ExecuteTemplate(&out, page, data))
// The name field and its hint, up to the end of their div.
_, field, found := strings.Cut(out.String(), `id="name"`)
require.True(t, found, page)
field, _, _ = strings.Cut(field, "</div>")
assert.Contains(t, field, `pattern="`+pattern+`"`, page)
assert.Contains(t, field, `minlength="`+strconv.Itoa(appNameMinLength)+`"`, page)
assert.Contains(t, field, `maxlength="`+strconv.Itoa(appNameMaxLength)+`"`, page)
assert.Contains(t, field, "hyphens, and dots", page)
}
}
+45
View File
@@ -0,0 +1,45 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/deploy"
)
// TestAppPageSaysFilesAreLostOnlyWhenAppHasNoVolumes checks that the app page
// shows deploy.NoVolumesWarning until the app gets a volume mount.
func TestAppPageSaysFilesAreLostOnlyWhenAppHasNoVolumes(t *testing.T) {
t.Parallel()
testCtx := setupTestHandlers(t)
createdApp := createTestApp(t, testCtx, "no-volumes-app")
renderAppPage := func() string {
request := httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/apps/"+createdApp.ID, nil,
)
request = addChiURLParams(request, map[string]string{"id": createdApp.ID})
recorder := httptest.NewRecorder()
testCtx.handlers.HandleAppDetail().ServeHTTP(recorder, request)
require.Equal(t, http.StatusOK, recorder.Code)
return recorder.Body.String()
}
assert.Contains(t, renderAppPage(), deploy.NoVolumesWarning)
volume := models.NewVolume(testCtx.database)
volume.AppID = createdApp.ID
volume.HostPath = "/srv/no-volumes-app"
volume.ContainerPath = "/data"
require.NoError(t, volume.Save(t.Context()))
assert.NotContains(t, renderAppPage(), deploy.NoVolumesWarning)
}
@@ -1,77 +0,0 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/service/app"
)
// TestDashboardTableFitsCard checks that the card around the app table
// scrolls sideways instead of hiding what does not fit, and that a long
// repository URL wraps instead of pushing the action buttons out.
func TestDashboardTableFitsCard(t *testing.T) {
t.Parallel()
testCtx := setupTestHandlers(t)
repoURL := "https://git.example.com/user/" +
"a-repository-name-long-enough-to-push-the-action-buttons-out.git"
_, err := testCtx.appSvc.CreateApp(t.Context(), app.CreateAppInput{
Name: "long-url-app",
RepoURL: repoURL,
})
require.NoError(t, err)
request := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
recorder := httptest.NewRecorder()
testCtx.handlers.HandleDashboard().ServeHTTP(recorder, request)
require.Equal(t, http.StatusOK, recorder.Code)
body := recorder.Body.String()
beforeTable, _, found := strings.Cut(body, `<table class="table">`)
require.True(t, found, "dashboard has no app table")
cardTag := beforeTable[strings.LastIndex(beforeTable, "<div"):]
assert.Contains(t, cardTag, "overflow-x-auto")
assert.NotContains(t, cardTag, "overflow-hidden")
beforeURL, _, found := strings.Cut(body, ">"+repoURL+"</td>")
require.True(t, found, "dashboard has no repository cell")
cellTag := beforeURL[strings.LastIndex(beforeURL, "<td"):]
assert.Contains(t, cellTag, "whitespace-normal")
assert.Contains(t, cellTag, "break-all")
}
// TestTopBarWrapsWhenNarrow checks that the top bar keeps a gap between the
// µPaaS title and the New App and Logout buttons, and puts the buttons on a
// second row in a window too narrow for one, instead of letting them meet.
func TestTopBarWrapsWhenNarrow(t *testing.T) {
t.Parallel()
testCtx := setupTestHandlers(t)
request := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
recorder := httptest.NewRecorder()
testCtx.handlers.HandleDashboard().ServeHTTP(recorder, request)
require.Equal(t, http.StatusOK, recorder.Code)
_, afterNav, found := strings.Cut(recorder.Body.String(), `<nav class="app-bar">`)
require.True(t, found, "dashboard has no top bar")
rowTag, _, _ := strings.Cut(strings.TrimSpace(afterNav), ">")
assert.Contains(t, rowTag, "flex-wrap")
assert.Contains(t, rowTag, "gap-")
}
-3
View File
@@ -109,9 +109,6 @@ func createAppServices(
})
require.NoError(t, authErr)
// 1 MiB per password hash instead of 64 MiB; see auth.Service.ArgonMemory.
authSvc.ArgonMemory = 1024
appSvc, appErr := app.New(fx.Lifecycle(nil), app.ServiceParams{
Logger: logInstance,
Database: dbInstance,
+1 -2
View File
@@ -4,7 +4,6 @@ package logger
import (
"log/slog"
"os"
"runtime"
"go.uber.org/fx"
@@ -82,6 +81,6 @@ func (l *Logger) Identify() {
l.log.Info("starting",
"appname", l.params.Globals.Appname,
"version", l.params.Globals.Version,
"arch", runtime.GOARCH,
"buildarch", l.params.Globals.Buildarch,
)
}
-34
View File
@@ -1,34 +0,0 @@
package logger //nolint:testpackage // sets the unexported log and params fields
import (
"bytes"
"encoding/json"
"log/slog"
"runtime"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/globals"
)
func TestIdentifyLogsArchitectureFromRuntime(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
l := &Logger{
log: slog.New(slog.NewJSONHandler(&buf, nil)),
params: Params{
Globals: &globals.Globals{Appname: "upaas-test", Version: "test"},
},
}
l.Identify()
var line map[string]any
require.NoError(t, json.Unmarshal(buf.Bytes(), &line))
assert.Equal(t, runtime.GOARCH, line["arch"])
}
+2 -42
View File
@@ -203,12 +203,11 @@ func (d *Deployment) insert(ctx context.Context) error {
func (d *Deployment) update(ctx context.Context) error {
query := `
UPDATE deployments SET
commit_sha = ?, image_id = ?, container_id = ?, status = ?, logs = ?,
finished_at = ?
image_id = ?, container_id = ?, status = ?, logs = ?, finished_at = ?
WHERE id = ?`
_, err := d.db.Exec(ctx, query,
d.CommitSHA, d.ImageID, d.ContainerID, d.Status, d.Logs, d.FinishedAt, d.ID,
d.ImageID, d.ContainerID, d.Status, d.Logs, d.FinishedAt, d.ID,
)
return err
@@ -296,45 +295,6 @@ func FindDeploymentsByAppID(
return deployments, nil
}
// FindDeploymentImageIDs returns the IDs of the images an app's
// deployments built or rolled back to.
func FindDeploymentImageIDs(
ctx context.Context,
deployDB *database.Database,
appID string,
) ([]string, error) {
rows, err := deployDB.Query(ctx, `
SELECT DISTINCT image_id FROM deployments
WHERE app_id = ? AND image_id IS NOT NULL`,
appID,
)
if err != nil {
return nil, fmt.Errorf("querying deployment image IDs: %w", err)
}
defer func() { _ = rows.Close() }()
var imageIDs []string
for rows.Next() {
var imageID string
scanErr := rows.Scan(&imageID)
if scanErr != nil {
return nil, fmt.Errorf("scanning deployment image ID: %w", scanErr)
}
imageIDs = append(imageIDs, imageID)
}
rowsErr := rows.Err()
if rowsErr != nil {
return nil, fmt.Errorf("iterating deployment image IDs: %w", rowsErr)
}
return imageIDs, nil
}
// LatestDeploymentForApp finds the most recent deployment for an app.
//
//nolint:nilnil // returning nil,nil is idiomatic for "not found" in Active Record
-27
View File
@@ -564,33 +564,6 @@ func TestDeploymentMarkFinished(t *testing.T) {
assert.True(t, found.FinishedAt.Valid)
}
// TestDeploymentSaveStoresCommitSetAfterInsert checks that a commit set on a
// deployment after it was first saved, as a manual deploy does once the clone
// reads it, is stored by the next save.
func TestDeploymentSaveStoresCommitSetAfterInsert(t *testing.T) {
t.Parallel()
testDB, cleanup := setupTestDB(t)
defer cleanup()
app := createTestApp(t, testDB)
deployment := models.NewDeployment(testDB)
deployment.AppID = app.ID
err := deployment.Save(context.Background())
require.NoError(t, err)
deployment.CommitSHA = sql.NullString{String: "abc123def456", Valid: true}
err = deployment.Save(context.Background())
require.NoError(t, err)
found, err := models.FindDeployment(context.Background(), testDB, deployment.ID)
require.NoError(t, err)
assert.Equal(t, "abc123def456", found.CommitSHA.String)
}
func TestDeploymentFindByAppID(t *testing.T) {
t.Parallel()
+6 -14
View File
@@ -59,13 +59,6 @@ type ServiceParams struct {
// Service provides authentication functionality.
type Service struct {
// ArgonMemory is the memory each argon2id hash takes, in KiB. New sets
// argonMemory, 64 MiB, and upaasd never changes it. Tests lower it, since
// many 64 MiB hashes at once under the race detector need more memory
// than a 4 GiB build machine has. A hash verifies only with the value it
// was made with.
ArgonMemory uint32
log *slog.Logger
db *database.Database
store *sessions.CookieStore
@@ -84,11 +77,10 @@ func New(_ fx.Lifecycle, params ServiceParams) (*Service, error) {
}
return &Service{
ArgonMemory: argonMemory,
log: params.Logger.Get(),
db: params.Database,
store: store,
params: &params,
log: params.Logger.Get(),
db: params.Database,
store: store,
params: &params,
}, nil
}
@@ -105,7 +97,7 @@ func (svc *Service) HashPassword(password string) (string, error) {
[]byte(password),
salt,
argonTime,
svc.ArgonMemory,
argonMemory,
argonThreads,
argonKeyLen,
)
@@ -140,7 +132,7 @@ func (svc *Service) VerifyPassword(hashedPassword, password string) bool {
[]byte(password),
salt,
argonTime,
svc.ArgonMemory,
argonMemory,
argonThreads,
argonKeyLen,
)
-19
View File
@@ -65,10 +65,6 @@ func setupTestService(t *testing.T) (*auth.Service, func()) {
})
require.NoError(t, err)
// 1 MiB per hash instead of 64 MiB; see Service.ArgonMemory. The tests
// that use setupAuthService keep 64 MiB.
svc.ArgonMemory = 1024
// t.TempDir() automatically cleans up after test
cleanup := func() {}
@@ -241,21 +237,6 @@ func TestVerifyPassword(testingT *testing.T) {
})
}
// TestHashPasswordWithUpaasdMemory hashes and verifies a password with the
// memory New sets, which upaasd uses. setupTestService lowers it.
func TestHashPasswordWithUpaasdMemory(t *testing.T) {
t.Parallel()
svc := setupAuthService(t, false)
require.Equal(t, uint32(64*1024), svc.ArgonMemory)
hash, err := svc.HashPassword("correctpassword")
require.NoError(t, err)
assert.True(t, svc.VerifyPassword(hash, "correctpassword"))
assert.False(t, svc.VerifyPassword(hash, "wrongpassword"))
}
func TestIsSetupRequired(testingT *testing.T) {
testingT.Parallel()
+30 -81
View File
@@ -56,6 +56,12 @@ var (
ErrNoPreviousImage = errors.New("no previous image available for rollback")
)
// NoVolumesWarning is shown on the page of an app with no volume mounts and
// written into each of its deploy logs.
const NoVolumesWarning = "This app has no volume mounts, so the files it writes " +
"are lost whenever a deploy or rollback replaces its container; " +
"a restart of the container keeps them."
// logFlushInterval is how often to flush buffered logs to the database.
const logFlushInterval = time.Second
@@ -369,6 +375,11 @@ func (svc *Service) Deploy(
svc.logWebhookPayload(bgCtx, deployment, webhookEvent)
volumes, err := app.GetVolumes(bgCtx)
if err == nil && len(volumes) == 0 {
_ = deployment.AppendLog(bgCtx, NoVolumesWarning)
}
err = svc.updateAppStatusBuilding(bgCtx, app)
if err != nil {
return err
@@ -735,103 +746,44 @@ func (svc *Service) recordDeployedImage(
return nil
}
// removeUnusedImages removes the app's images except those an app's running
// container uses or its Rollback would start. Docker deletes a tagged image
// only once no other tag, such as another app's, and no container still
// uses it.
// removeUnusedImages removes the app's tags (upaas-<app>:<deployment>, set by
// buildImage) except those of the image the running container uses and the
// one Rollback would start. Docker deletes an image only once no other tag,
// such as another app's, and no container still uses it.
func (svc *Service) removeUnusedImages(
ctx context.Context,
app *models.App,
deployment *models.Deployment,
) {
images, err := svc.findAppImages(ctx, app)
tags, err := svc.docker.ListImageTags(ctx, "upaas-"+app.Name)
if err != nil {
svc.log.Error("failed to list app images", "error", err, "app", app.Name)
return
}
keep, err := svc.imagesToKeep(ctx)
if err != nil {
svc.log.Error("failed to list the images apps use", "error", err, "app", app.Name)
return
}
for _, name := range slices.Sorted(maps.Keys(images)) {
if keep[images[name].String()] {
for _, tag := range slices.Sorted(maps.Keys(tags)) {
imageID := tags[tag].String()
if imageID == app.ImageID.String || imageID == app.PreviousImageID.String {
continue
}
removeErr := svc.docker.RemoveImageTag(ctx, name)
removeErr := svc.docker.RemoveImageTag(ctx, tag)
if removeErr != nil {
svc.log.Error("failed to remove old image",
"error", removeErr, "app", app.Name, "image", name)
"error", removeErr, "app", app.Name, "tag", tag)
_ = deployment.AppendLog(
ctx,
"WARNING: failed to remove old image "+name+": "+removeErr.Error(),
"WARNING: failed to remove old image "+tag+": "+removeErr.Error(),
)
continue
}
_ = deployment.AppendLog(ctx, "Removed old image: "+name)
_ = deployment.AppendLog(ctx, "Removed old image: "+tag)
}
}
// findAppImages returns the app's images, each under the name it is removed
// by: its tag, upaas-<app>:<short hash> as set by buildImage, or its ID if
// it has none. A redeploy of a commit gives the commit's tag to the new
// image, so the old one is found among the images the app's deployments
// recorded.
func (svc *Service) findAppImages(
ctx context.Context,
app *models.App,
) (map[string]docker.ImageID, error) {
images, err := svc.docker.ListImageTags(ctx, "upaas-"+app.Name)
if err != nil {
return nil, fmt.Errorf("failed to list image tags: %w", err)
}
untagged, err := svc.docker.ListUntaggedImages(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list untagged images: %w", err)
}
recorded, err := models.FindDeploymentImageIDs(ctx, svc.db, app.ID)
if err != nil {
return nil, fmt.Errorf("failed to find deployment images: %w", err)
}
for _, imageID := range untagged {
if slices.Contains(recorded, imageID.String()) {
images[imageID.String()] = imageID
}
}
return images, nil
}
// imagesToKeep returns the IDs of the image each app's running container
// uses and the one its Rollback would start. It covers every app because
// apps that build the same commit can share an image, and a redeploy can
// take the tag that kept the image for one of them.
func (svc *Service) imagesToKeep(ctx context.Context) (map[string]bool, error) {
apps, err := models.AllApps(ctx, svc.db)
if err != nil {
return nil, fmt.Errorf("failed to list apps: %w", err)
}
keep := make(map[string]bool)
for _, app := range apps {
keep[app.ImageID.String] = true
keep[app.PreviousImageID.String] = true
}
return keep, nil
}
// cleanupCancelledDeploy removes orphan resources left by a cancelled deployment.
func (svc *Service) cleanupCancelledDeploy(
ctx context.Context,
@@ -971,14 +923,14 @@ func (svc *Service) buildImage(
app *models.App,
deployment *models.Deployment,
) (docker.ImageID, error) {
workDir, shortSHA, cleanup, err := svc.cloneRepository(ctx, app, deployment)
workDir, cleanup, err := svc.cloneRepository(ctx, app, deployment)
if err != nil {
return "", err
}
defer cleanup()
imageTag := "upaas-" + app.Name + ":" + shortSHA
imageTag := fmt.Sprintf("upaas-%s:%d", app.Name, deployment.ID)
// Create log writer that flushes build output to deployment logs every second
logWriter := newDeploymentLogWriter(ctx, deployment)
@@ -1013,14 +965,11 @@ func (svc *Service) buildImage(
return imageID, nil
}
// cloneRepository clones the app's repository for a build. It returns the
// directory of the clone, git's short form of the commit checked out, and a
// function that removes the clone.
func (svc *Service) cloneRepository(
ctx context.Context,
app *models.App,
deployment *models.Deployment,
) (string, string, func(), error) {
) (string, func(), error) {
// Use a subdirectory of DataDir for builds since it's mounted from the host
// and accessible to Docker for bind mounts (unlike /tmp inside the container).
// Structure: builds/<appname>/<deployment-id>-<random>/
@@ -1037,7 +986,7 @@ func (svc *Service) cloneRepository(
fmt.Errorf("failed to create builds dir: %w", err),
)
return "", "", nil, fmt.Errorf("failed to create builds dir: %w", err)
return "", nil, fmt.Errorf("failed to create builds dir: %w", err)
}
buildDir, err := os.MkdirTemp(appBuildsDir, fmt.Sprintf("%d-*", deployment.ID))
@@ -1049,7 +998,7 @@ func (svc *Service) cloneRepository(
fmt.Errorf("failed to create temp dir: %w", err),
)
return "", "", nil, fmt.Errorf("failed to create temp dir: %w", err)
return "", nil, fmt.Errorf("failed to create temp dir: %w", err)
}
cleanup := func() { _ = os.RemoveAll(buildDir) }
@@ -1085,7 +1034,7 @@ func (svc *Service) cloneRepository(
fmt.Errorf("failed to clone repo: %w", cloneErr),
)
return "", "", nil, fmt.Errorf("failed to clone repo: %w", cloneErr)
return "", nil, fmt.Errorf("failed to clone repo: %w", cloneErr)
}
svc.processCloneResult(ctx, app, deployment, cloneResult, commitSHA)
@@ -1093,7 +1042,7 @@ func (svc *Service) cloneRepository(
// Return the 'work' subdirectory where the repo was cloned
workDir := filepath.Join(buildDir, "work")
return workDir, cloneResult.ShortSHA, cleanup, nil
return workDir, cleanup, nil
}
// processCloneResult handles the result of a git clone operation.
@@ -1,121 +0,0 @@
package deploy_test
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"os"
"slices"
"strings"
"testing"
"github.com/distribution/reference"
"github.com/docker/docker/daemon/names"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/database"
"sneak.berlin/go/upaas/internal/globals"
"sneak.berlin/go/upaas/internal/handlers"
"sneak.berlin/go/upaas/internal/logger"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/app"
)
// TestDeployAppWithDotInName creates an app named sneak.berlin through the
// new app form, as a user does, then builds and deploys it against a fake
// Docker API and checks that Docker accepts the names of the image it
// builds and of the container it runs, using Docker's own rules for each.
func TestDeployAppWithDotInName(t *testing.T) {
t.Parallel()
api := &fakeImageAPI{
images: map[string][]string{},
shortSHA: "abc1234",
nextID: "sha256:built",
}
svc, db := newImageTestService(t, api)
ctx := context.Background()
createdApp := createAppWithForm(t, db, "sneak.berlin")
deployment := models.NewDeployment(db)
deployment.AppID = createdApp.ID
require.NoError(t, deployment.Save(ctx))
imageID, err := svc.BuildImage(ctx, createdApp, deployment)
require.NoError(t, err)
require.NoError(t, svc.DeployContainer(ctx, createdApp, deployment, imageID))
images, _ := api.state()
api.mu.Lock()
containers := slices.Clone(api.containers)
api.mu.Unlock()
require.Equal(t, map[string][]string{
"sha256:built": {"upaas-sneak.berlin:abc1234"},
}, images)
_, err = reference.ParseNormalizedNamed("upaas-sneak.berlin:abc1234")
require.NoError(t, err)
require.Equal(t, []string{"upaas-sneak.berlin"}, containers)
assert.Regexp(t, names.RestrictedNamePattern, containers[0])
assert.DirExists(t, svc.GetBuildDirExported(createdApp.Name))
}
// createAppWithForm posts the new app form with the given name, which
// checks the name as it does for a user, and returns the app it created.
func createAppWithForm(
t *testing.T,
db *database.Database,
name string,
) *models.App {
t.Helper()
log := logger.NewForTest(slog.New(slog.NewTextHandler(os.Stderr, nil)))
appSvc, err := app.New(nil, app.ServiceParams{Logger: log, Database: db})
require.NoError(t, err)
globalInstance, err := globals.New(nil)
require.NoError(t, err)
handlersInstance, err := handlers.New(nil, handlers.Params{
Logger: log,
Globals: globalInstance,
Database: db,
App: appSvc,
})
require.NoError(t, err)
form := url.Values{
"name": {name},
"repo_url": {"git@example.com:sneak/" + name + ".git"},
}
request := httptest.NewRequestWithContext(
t.Context(), http.MethodPost, "/apps", strings.NewReader(form.Encode()),
)
request.Header.Set("Content-Type", "application/x-www-form-urlencoded")
recorder := httptest.NewRecorder()
handlersInstance.HandleAppCreate().ServeHTTP(recorder, request)
// The form redirects to the new app's page; it shows the form again,
// with the error, when it refuses the name.
require.Equal(t, http.StatusSeeOther, recorder.Code, recorder.Body.String())
appID, found := strings.CutPrefix(recorder.Header().Get("Location"), "/apps/")
require.True(t, found)
createdApp, err := models.FindApp(t.Context(), db, appID)
require.NoError(t, err)
require.NotNil(t, createdApp)
return createdApp
}
+1 -3
View File
@@ -34,8 +34,6 @@ func TestBuildImageLogsBuildErrorBeforeDeployError(t *testing.T) {
switch {
case strings.HasSuffix(r.URL.Path, "/containers/create"):
_, _ = w.Write([]byte(`{"Id":"gitcontainer"}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w, "abc1234")
case strings.HasSuffix(r.URL.Path, "/version"):
_, _ = w.Write([]byte(`{"Version":"27.3.1","ApiVersion":"1.47"}`))
case strings.HasSuffix(r.URL.Path, "/build"):
@@ -79,7 +77,7 @@ func TestBuildImageLogsBuildErrorBeforeDeployError(t *testing.T) {
// The service has no notify service: the app has no ntfy topic and no
// Slack webhook, so the build failure notification sends nothing.
svc := deploy.NewTestServiceWithConfig(log, cfg, db, dockerClient)
svc := deploy.NewTestServiceWithConfig(log, cfg, dockerClient)
_, err = svc.BuildImage(ctx, app, deployment)
require.EqualError(t, err, "failed to build image: exit code: 1")
@@ -20,7 +20,7 @@ func TestCleanupCancelledDeploy_RemovesBuildDir(t *testing.T) {
tmpDir := t.TempDir()
cfg := &config.Config{DataDir: tmpDir}
svc := deploy.NewTestServiceWithConfig(slog.Default(), cfg, nil, nil)
svc := deploy.NewTestServiceWithConfig(slog.Default(), cfg, nil)
// Create a fake build directory matching the deployment pattern
appName := "test-app"
@@ -59,7 +59,7 @@ func TestCleanupCancelledDeploy_NoBuildDir(t *testing.T) {
tmpDir := t.TempDir()
cfg := &config.Config{DataDir: tmpDir}
svc := deploy.NewTestServiceWithConfig(slog.Default(), cfg, nil, nil)
svc := deploy.NewTestServiceWithConfig(slog.Default(), cfg, nil)
// Should not panic when build dir doesn't exist
svc.CleanupCancelledDeploy(context.Background(), "nonexistent-app", 1, "")
+46 -326
View File
@@ -3,21 +3,14 @@ package deploy_test
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"io"
"log/slog"
"maps"
"net/http"
"net/http/httptest"
"os"
"slices"
"strings"
"sync"
"testing"
"github.com/docker/docker/api/types/image"
"github.com/docker/docker/pkg/stdcopy"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
@@ -30,145 +23,45 @@ import (
"sneak.berlin/go/upaas/internal/service/deploy"
)
// fakeImageAPI is a fake Docker API that keeps images and their tags as
// Docker does: a build gives its tag to the image it builds, and removing
// an image's last tag, or an untagged image by its ID, deletes the image.
// It also answers the steps of a git clone that reports shortSHA.
type fakeImageAPI struct {
mu sync.Mutex
images map[string][]string // image ID -> tags
shortSHA string // the commit's short hash the clone reports
nextID string // ID of the image the next build creates
removed []string // each tag or ID removed
forced bool // whether a removal was forced
containers []string // name of each named container created
}
// TestRecordDeployedImageRemovesOldImages runs the step after a deploy
// against a fake Docker API. Image one is also tagged for another app,
// image two was the previous image, three the current one, four is new.
func TestRecordDeployedImageRemovesOldImages(t *testing.T) {
t.Parallel()
func (api *fakeImageAPI) ServeHTTP(w http.ResponseWriter, r *http.Request) {
api.mu.Lock()
defer api.mu.Unlock()
var (
mu sync.Mutex
removed []string
forced bool
)
w.Header().Set("Content-Type", "application/json")
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, name, isImage := strings.Cut(r.URL.Path, "/images/")
switch {
case r.Method == http.MethodDelete:
_, name, _ := strings.Cut(r.URL.Path, "/images/")
switch {
case strings.HasSuffix(r.URL.Path, "/images/json"):
dangling := strings.Contains(r.URL.Query().Get("filters"), "dangling")
api.listImages(w, dangling)
case isImage && r.Method == http.MethodDelete:
api.forced = api.forced || r.URL.Query().Get("force") != ""
api.removeImage(w, name)
case isImage && strings.HasSuffix(name, "/json"):
api.inspectImage(w, strings.TrimSuffix(name, "/json"))
case strings.HasSuffix(r.URL.Path, "/build"):
tag := r.URL.Query().Get("t")
api.untag(tag)
api.images[api.nextID] = append(api.images[api.nextID], tag)
case strings.HasSuffix(r.URL.Path, "/version"):
_, _ = w.Write([]byte(`{"Version":"27.3.1","ApiVersion":"1.47"}`))
case strings.HasSuffix(r.URL.Path, "/containers/create"):
// The git clone's container has no name; the app's has.
if containerName := r.URL.Query().Get("name"); containerName != "" {
api.containers = append(api.containers, containerName)
}
mu.Lock()
_, _ = w.Write([]byte(`{"Id":"gitcontainer"}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w, api.shortSHA)
default:
// The other steps of the git clone, which succeeds.
_, _ = w.Write([]byte(`{}`))
}
}
removed = append(removed, name)
forced = forced || r.URL.Query().Get("force") != ""
mu.Unlock()
// listImages lists the untagged images, or else the tagged ones.
func (api *fakeImageAPI) listImages(w http.ResponseWriter, dangling bool) {
list := []image.Summary{}
for _, id := range slices.Sorted(maps.Keys(api.images)) {
if (len(api.images[id]) == 0) == dangling {
list = append(list, image.Summary{ID: id, RepoTags: api.images[id]})
}
}
data, err := json.Marshal(list)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
_, _ = w.Write(data)
}
func (api *fakeImageAPI) inspectImage(w http.ResponseWriter, name string) {
for id, tags := range api.images {
if id == name || slices.Contains(tags, name) {
_, _ = fmt.Fprintf(w, `{"Id":%q}`, id)
return
}
}
w.WriteHeader(http.StatusNotFound)
_, _ = w.Write([]byte(`{"message":"No such image"}`))
}
func (api *fakeImageAPI) removeImage(w http.ResponseWriter, name string) {
api.removed = append(api.removed, name)
id := name
if _, isID := api.images[name]; !isID {
id = api.untag(name)
}
if len(api.images[id]) == 0 {
delete(api.images, id)
}
_, _ = w.Write([]byte(`[]`))
}
// untag removes tag from the image that has it, leaving the image, and
// returns the image's ID.
func (api *fakeImageAPI) untag(tag string) string {
for id, tags := range api.images {
if slices.Contains(tags, tag) {
api.images[id] = slices.DeleteFunc(tags, func(t string) bool { return t == tag })
return id
}
}
return ""
}
// state returns each image's tags and each tag or ID removed.
func (api *fakeImageAPI) state() (map[string][]string, []string) {
api.mu.Lock()
defer api.mu.Unlock()
return maps.Clone(api.images), slices.Clone(api.removed)
}
// writeCloneOutput writes the line of a git clone's output that gives the
// commit's short hash, as Docker sends a container's log: after a header.
func writeCloneOutput(w io.Writer, shortSHA string) {
out := stdcopy.NewStdWriter(w, stdcopy.Stdout)
_, _ = fmt.Fprintf(out, "SHORT_SHA:%s\n", shortSHA)
}
// newImageTestService returns a deploy service that uses api as its
// Docker API, and its database.
func newImageTestService(
t *testing.T,
api *fakeImageAPI,
) (*deploy.Service, *database.Database) {
t.Helper()
srv := httptest.NewServer(api)
_, _ = w.Write([]byte(`[]`))
case strings.HasSuffix(r.URL.Path, "/images/json"):
_, _ = w.Write([]byte(`[
{"Id":"sha256:one","RepoTags":["upaas-myapp:1","upaas-otherapp:7"]},
{"Id":"sha256:two","RepoTags":["upaas-myapp:2"]},
{"Id":"sha256:three","RepoTags":["upaas-myapp:3"]},
{"Id":"sha256:four","RepoTags":["upaas-myapp:4"]}
]`))
default:
_, _ = w.Write([]byte(`{}`))
}
},
))
t.Cleanup(srv.Close)
log := slog.New(slog.NewTextHandler(os.Stderr, nil))
@@ -184,203 +77,30 @@ func newImageTestService(
t.Cleanup(lifecycle.RequireStop)
db := database.NewTestDatabase(t)
dataDir := t.TempDir()
cfg := &config.Config{DataDir: dataDir, HostDataDir: dataDir}
return deploy.NewTestServiceWithConfig(log, cfg, db, dockerClient), db
}
// saveApp saves an app with the given current and previous image.
func saveApp(
t *testing.T,
db *database.Database,
name, imageID, previousImageID string,
) *models.App {
t.Helper()
app := models.NewApp(db)
app.ID = name + "-id"
app.Name = name
app.ImageID = sql.NullString{String: imageID, Valid: true}
app.PreviousImageID = sql.NullString{String: previousImageID, Valid: true}
require.NoError(t, app.Save(context.Background()))
return app
}
// TestRecordDeployedImageRemovesOldImages runs the step after a deploy
// against a fake Docker API. Image one has a tag from before images were
// tagged with their commit, and is also tagged for another app. Image two
// was the previous image, three the current one, four is new. Image five is
// the other app's previous image, which a redeploy of its commit left
// without the other app's tag.
func TestRecordDeployedImageRemovesOldImages(t *testing.T) {
t.Parallel()
api := &fakeImageAPI{images: map[string][]string{
"sha256:one": {"upaas-myapp:140", "upaas-otherapp:1a2b3c4"},
"sha256:two": {"upaas-myapp:2b3c4d5"},
"sha256:three": {"upaas-myapp:3c4d5e6"},
"sha256:four": {"upaas-myapp:4d5e6f7"},
"sha256:five": {"upaas-myapp:5e6f7a8"},
}}
svc, db := newImageTestService(t, api)
ctx := context.Background()
app := saveApp(t, db, "myapp", "sha256:three", "sha256:two")
saveApp(t, db, "otherapp", "sha256:other", "sha256:five")
app := models.NewApp(db)
app.ID = "myapp-id"
app.Name = "myapp"
app.ImageID = sql.NullString{String: "sha256:three", Valid: true}
app.PreviousImageID = sql.NullString{String: "sha256:two", Valid: true}
require.NoError(t, app.Save(ctx))
deployment := models.NewDeployment(db)
deployment.AppID = app.ID
require.NoError(t, deployment.Save(ctx))
err := svc.RecordDeployedImage(ctx, app, deployment, "sha256:four")
svc := deploy.NewTestServiceWithConfig(log, &config.Config{}, dockerClient)
err = svc.RecordDeployedImage(ctx, app, deployment, "sha256:four")
require.NoError(t, err)
assert.Equal(t, "sha256:four", app.ImageID.String)
assert.Equal(t, "sha256:three", app.PreviousImageID.String)
images, removed := api.state()
mu.Lock()
defer mu.Unlock()
assert.Equal(t, []string{"upaas-myapp:140", "upaas-myapp:2b3c4d5"}, removed)
assert.Equal(t, map[string][]string{
"sha256:one": {"upaas-otherapp:1a2b3c4"},
"sha256:three": {"upaas-myapp:3c4d5e6"},
"sha256:four": {"upaas-myapp:4d5e6f7"},
"sha256:five": {"upaas-myapp:5e6f7a8"},
}, images)
assert.False(t, api.forced, "old images must be removed without force")
}
// TestRecordDeployedImageRemovesOnlyTheAppsUntaggedImages runs the step after
// a deploy against a fake Docker API that holds three untagged images: one an
// earlier deployment of the app recorded, one a deployment of another app
// recorded, and one no deployment recorded, such as an image upaas never
// built. Only the app's own is removed.
func TestRecordDeployedImageRemovesOnlyTheAppsUntaggedImages(t *testing.T) {
t.Parallel()
api := &fakeImageAPI{images: map[string][]string{
"sha256:new": {"upaas-myapp:1a2b3c4"},
"sha256:myapp": {},
"sha256:otherapp": {},
"sha256:unknown": {},
}}
svc, db := newImageTestService(t, api)
ctx := context.Background()
app := saveApp(t, db, "myapp", "", "")
otherApp := saveApp(t, db, "otherapp", "", "")
saveDeployment := func(appID, imageID string) *models.Deployment {
t.Helper()
deployment := models.NewDeployment(db)
deployment.AppID = appID
deployment.ImageID = sql.NullString{String: imageID, Valid: true}
require.NoError(t, deployment.Save(ctx))
return deployment
}
saveDeployment(app.ID, "sha256:myapp")
saveDeployment(otherApp.ID, "sha256:otherapp")
deployment := saveDeployment(app.ID, "sha256:new")
err := svc.RecordDeployedImage(ctx, app, deployment, "sha256:new")
require.NoError(t, err)
images, removed := api.state()
assert.Equal(t, []string{"sha256:myapp"}, removed)
assert.Equal(t, map[string][]string{
"sha256:new": {"upaas-myapp:1a2b3c4"},
"sha256:otherapp": {},
"sha256:unknown": {},
}, images)
}
// TestRedeployRemovesImagesLeftWithoutTag deploys commits against a fake
// Docker API, some of them again. A build takes the commit's tag from the
// image an earlier build of it made. That image is kept, without a tag,
// while the app runs it or Rollback would start it, and is removed by its
// ID once neither does.
func TestRedeployRemovesImagesLeftWithoutTag(t *testing.T) {
t.Parallel()
const (
tagABC1234 = "upaas-myapp:abc1234"
tag0123ABC = "upaas-myapp:0123abc"
retriedImage = "sha256:retried-0123abc"
)
// The app runs commit abc1234 and would roll back to def5678. The last
// deploy, of commit 0123abc, failed after its build.
api := &fakeImageAPI{images: map[string][]string{
"sha256:built-abc1234": {tagABC1234},
"sha256:built-def5678": {"upaas-myapp:def5678"},
"sha256:failed-0123abc": {tag0123ABC},
}}
svc, db := newImageTestService(t, api)
ctx := context.Background()
app := saveApp(t, db, "myapp", "sha256:built-abc1234", "sha256:built-def5678")
for imageID := range api.images {
deployment := models.NewDeployment(db)
deployment.AppID = app.ID
deployment.ImageID = sql.NullString{String: imageID, Valid: true}
require.NoError(t, deployment.Save(ctx))
}
deployCommit := func(shortSHA, imageID string) {
t.Helper()
api.mu.Lock()
api.shortSHA = shortSHA
api.nextID = imageID
api.mu.Unlock()
deployment := models.NewDeployment(db)
deployment.AppID = app.ID
require.NoError(t, deployment.Save(ctx))
built, err := svc.BuildImage(ctx, app, deployment)
require.NoError(t, err)
require.NoError(t, svc.RecordDeployedImage(ctx, app, deployment, built))
}
// The failed deploy's image loses its tag, and nothing uses it.
deployCommit("0123abc", retriedImage)
images, _ := api.state()
assert.Equal(t, map[string][]string{
"sha256:built-abc1234": {tagABC1234},
retriedImage: {tag0123ABC},
}, images)
// The running image loses its tag and becomes the one Rollback starts.
deployCommit("0123abc", "sha256:rebuilt-0123abc")
images, _ = api.state()
assert.Equal(t, map[string][]string{
"sha256:rebuilt-0123abc": {tag0123ABC},
retriedImage: {},
}, images)
assert.Equal(t, retriedImage, app.PreviousImageID.String)
// Once Rollback no longer needs it, the untagged image is removed.
deployCommit("4567def", "sha256:built-4567def")
images, removed := api.state()
assert.Equal(t, map[string][]string{
"sha256:built-4567def": {"upaas-myapp:4567def"},
"sha256:rebuilt-0123abc": {tag0123ABC},
}, images)
assert.Equal(t, []string{
"sha256:failed-0123abc", "upaas-myapp:def5678", // first deploy
tagABC1234, // second deploy
retriedImage, // third deploy
}, removed)
assert.False(t, api.forced, "old images must be removed without force")
assert.Equal(t, []string{"upaas-myapp:1", "upaas-myapp:2"}, removed)
assert.False(t, forced, "old image tags must be removed without force")
}
@@ -0,0 +1,79 @@
package deploy_test
import (
"context"
"log/slog"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/config"
"sneak.berlin/go/upaas/internal/database"
"sneak.berlin/go/upaas/internal/docker"
"sneak.berlin/go/upaas/internal/logger"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/deploy"
"sneak.berlin/go/upaas/internal/service/notify"
)
// deployLog deploys a new app, with one volume mount when withVolume is set,
// and returns the deploy's log. Docker is not connected, so the deploy fails
// at the git clone, after the start of its log is written.
func deployLog(t *testing.T, withVolume bool) string {
t.Helper()
log := logger.NewForTest(slog.New(slog.NewTextHandler(os.Stderr, nil)))
dataDir := t.TempDir()
cfg := &config.Config{DataDir: dataDir, HostDataDir: dataDir}
db := database.NewTestDatabase(t)
dockerClient, err := docker.New(nil, docker.Params{Logger: log, Config: cfg})
require.NoError(t, err)
notifySvc, err := notify.New(nil, notify.ServiceParams{Logger: log})
require.NoError(t, err)
svc, err := deploy.New(nil, deploy.ServiceParams{
Logger: log, Config: cfg, Database: db,
Docker: dockerClient, Notify: notifySvc,
})
require.NoError(t, err)
ctx := context.Background()
app := models.NewApp(db)
app.ID = "volumesapp-id"
app.Name = "volumesapp"
app.Branch = "main"
require.NoError(t, app.Save(ctx))
if withVolume {
volume := models.NewVolume(db)
volume.AppID = app.ID
volume.HostPath = "/srv/volumesapp"
volume.ContainerPath = "/data"
require.NoError(t, volume.Save(ctx))
}
err = svc.Deploy(ctx, app, nil, false)
require.ErrorIs(t, err, docker.ErrNotConnected)
deployments, err := app.GetDeployments(ctx, 1)
require.NoError(t, err)
require.Len(t, deployments, 1)
return deployments[0].Logs.String
}
func TestDeployLogSaysFilesAreLostOnlyWhenAppHasNoVolumes(t *testing.T) {
t.Parallel()
logWithoutVolumes := deployLog(t, false)
assert.Equal(t, 1, strings.Count(logWithoutVolumes, deploy.NoVolumesWarning),
logWithoutVolumes)
assert.NotContains(t, deployLog(t, true), deploy.NoVolumesWarning)
}
+1 -15
View File
@@ -9,7 +9,6 @@ import (
"strings"
"sneak.berlin/go/upaas/internal/config"
"sneak.berlin/go/upaas/internal/database"
"sneak.berlin/go/upaas/internal/docker"
"sneak.berlin/go/upaas/internal/models"
)
@@ -45,18 +44,15 @@ func (svc *Service) UnlockApp(appID string) {
svc.unlockApp(appID)
}
// NewTestServiceWithConfig creates a Service with config, database and
// docker client for testing.
// NewTestServiceWithConfig creates a Service with config and docker client for testing.
func NewTestServiceWithConfig(
log *slog.Logger,
cfg *config.Config,
db *database.Database,
dockerClient *docker.Client,
) *Service {
return &Service{
log: log,
config: cfg,
db: db,
docker: dockerClient,
}
}
@@ -113,16 +109,6 @@ func (svc *Service) BuildImage(
return svc.buildImage(ctx, app, deployment)
}
// DeployContainer exposes deployContainerWithTimeout for testing.
func (svc *Service) DeployContainer(
ctx context.Context,
app *models.App,
deployment *models.Deployment,
imageID docker.ImageID,
) error {
return svc.deployContainerWithTimeout(ctx, app, deployment, imageID)
}
// BuildContainerOptionsExported exposes buildContainerOptions for testing.
func (svc *Service) BuildContainerOptionsExported(
ctx context.Context,
+2
View File
@@ -318,6 +318,8 @@
</tbody>
</table>
</div>
{{else}}
<p class="alert-warning">{{.NoVolumesWarning}}</p>
{{end}}
<form method="POST" action="/apps/{{.App.ID}}/volumes" class="flex flex-col sm:flex-row gap-2 items-end">
{{ .CSRFField }}
+2 -4
View File
@@ -30,12 +30,10 @@
name="name"
value="{{.App.Name}}"
required
minlength="2"
maxlength="63"
pattern="[a-z0-9]+((\.|-+)[a-z0-9]+)*"
pattern="[a-z0-9-]+"
class="input"
>
<p class="text-sm text-gray-500 mt-1">Lowercase letters, numbers, hyphens, and dots, such as my-app or example.com</p>
<p class="text-sm text-gray-500 mt-1">Lowercase letters, numbers, and hyphens only</p>
</div>
<div class="form-group">
+2 -4
View File
@@ -30,13 +30,11 @@
name="name"
value="{{.Name}}"
required
minlength="2"
maxlength="63"
pattern="[a-z0-9]+((\.|-+)[a-z0-9]+)*"
pattern="[a-z0-9-]+"
class="input"
placeholder="my-app"
>
<p class="text-sm text-gray-500 mt-1">Lowercase letters, numbers, hyphens, and dots, such as my-app or example.com</p>
<p class="text-sm text-gray-500 mt-1">Lowercase letters, numbers, and hyphens only</p>
</div>
<div class="form-group">
+1 -1
View File
@@ -26,7 +26,7 @@
{{define "nav"}}
<nav class="app-bar">
<div class="max-w-6xl mx-auto flex flex-wrap justify-between items-center gap-3">
<div class="max-w-6xl mx-auto flex justify-between items-center">
<div class="flex items-center gap-3">
<a href="/" class="text-xl font-medium text-gray-900 hover:text-primary-600 transition-colors">µPaaS</a>
<span class="text-sm text-gray-500">by <a href="https://sneak.berlin" class="text-primary-600 hover:text-primary-800">@sneak</a></span>
+2 -2
View File
@@ -20,7 +20,7 @@
</div>
{{if .AppStats}}
<div class="card overflow-x-auto">
<div class="card overflow-hidden">
<table class="table">
<thead class="table-header">
<tr>
@@ -41,7 +41,7 @@
{{.App.Name}}
</a>
</td>
<td class="text-gray-500 font-mono text-xs whitespace-normal break-all">{{.App.RepoURL}}</td>
<td class="text-gray-500 font-mono text-xs">{{.App.RepoURL}}</td>
<td class="text-gray-500">{{.App.Branch}}</td>
<td>
{{if eq .App.Status "running"}}