4 Commits
Author SHA1 Message Date
sneak 1999ca6ba6 Tag built images with the commit's short hash (closes #239)
Check / check (pull_request) Skipped
Builds are tagged upaas-<app>:<short hash>, git's short form of
the commit checked out, instead of the deployment number. The clone
prints the short hash and fails without one.

A redeploy of a commit gives the tag to the new image. The cleanup
after a deploy now also finds the app's untagged images among those
its deployments recorded, and removes them by ID once the app neither
runs them nor would roll back to them. It keeps every image any app
uses, since apps that build the same commit can share one.

commit_sha is now saved on update so manual deploys keep the commit
read from the clone.

Model: opus-5-5
2026-10-01 19:36:31 +00:00
clawbot cc786a1e84 Say that a deploy keeps only an app's volumes when the app has none (closes #248)
Check / check (pull_request) Successful in 3m50s
An app with no volume mounts now shows, in its Volume Mounts section,
that the files it writes are lost whenever a deploy or rollback replaces
its container and that a restart keeps them. Each deploy of such an app
writes the same sentence into its log, after the webhook payload and
before the clone. The README says in one sentence that a deploy or
rollback starts a new container that keeps only the files in the app's
volume mounts.

Model: opus-5-5
2026-10-01 21:34:29 +02:00
clawbot 5f9948d7e2 Attach a BuildKit session to builds and demultiplex container logs (closes #251)
Check / check (pull_request) Successful in 3m59s
Builds now attach a BuildKit session over the Docker API, as the docker
command line does, and pass its ID with the build. BuildKit fetches a base
image that is not on the host through that session; without one, Docker
Engine 27 failed the build with "no active sessions". The session is
closed when the build ends.

Container logs are now read with stdcopy, so the clone output in the build
log and the app logs no longer carry Docker's 8-byte frame headers, and the
commit is read from the clone output; the header in front of the COMMIT
line kept it from being found.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-10-01 21:15:06 +02:00
clawbot d69f11f74c Have Docker create a missing volume host path (closes #235)
Check / check (pull_request) Successful in 4m54s
Docker refused to bind-mount a host path that did not exist, so an app's
first deploy failed until someone ran mkdir on the host. buildMounts now
sets BindOptions.CreateMountpoint on every bind mount, so the Docker
daemon creates a missing host path when the container starts and leaves
an existing one alone. upaas itself cannot create it: it runs in a
container and does not see the host's filesystem. The README gains a
Volume mounts section saying so. Needs Docker Engine 23.0 or later.

Model: opus-5-5
2026-10-01 20:48:13 +02:00
10 changed files with 427 additions and 74 deletions
+10
View File
@@ -271,6 +271,16 @@ names Dockerfile frontend 1.3 or later, such as `docker/dockerfile:1`.
Session secrets are automatically generated on first startup and persisted to
`$UPAAS_DATA_DIR/session.key`.
### Volume mounts
An app's volume mounts are bind mounts of host paths. When a host path does not
exist yet, upaas has Docker create it as an empty directory, owned by root, when
the app's container starts; there is no need to create it first. An existing
host path is left as it is. This needs Docker Engine 23.0 or later.
A deploy or rollback replaces the app's container with a new one, which keeps
only the files the app wrote to its volume mounts.
## License
WTFPL
+20 -3
View File
@@ -20,14 +20,31 @@ regress.
# Completed Steps
- 2026-10-01: An app with no volume mounts says on its page, and in the log of
each deploy, that a deploy or rollback loses the files it writes and a restart
keeps them; the README's Volume mounts section says a deploy or rollback keeps
only the files the app wrote to its volume mounts (#248).
- 2026-10-01: Builds attach a BuildKit session, as the docker command line does,
so a base image that is not on the host is pulled instead of the build failing
with "no active sessions" on Docker Engine 27. Container logs, and so the
clone output in the build log and the app logs, no longer carry Docker's
stream frame headers, and the commit is now read from the clone output (#251).
- 2026-10-01: An app's first deploy no longer fails when a volume's host path
does not exist yet: upaas asks Docker to create a missing host path when the
app's container starts and to leave an existing one alone, so nobody has to
create it on the host first. The README has a Volume mounts section saying so
(#235).
- 2026-09-29: Built images are tagged `upaas-<app>:<short hash>`, git's short
form of the commit built, instead of the deployment number. A redeploy of a
commit gives its tag to the new image; the old one is kept while the app runs
it or Rollback would start it, then removed by its ID, found among the images
the app's deployments recorded. The removal of old images now keeps every
image any app runs or would roll back to. The clone now reads the commit from
git's output, which Docker's log headers had hidden, so manual deploys record
their commit too (#239).
image any app runs or would roll back to. A deployment's commit is now saved
when it is updated, so manual deploys keep the commit read from the clone
(#239).
- 2026-09-29: The app page is 50% wider on large screens (84rem instead of
56rem), its build log and container log boxes are twice as tall, the build log
+58 -29
View File
@@ -3,12 +3,14 @@ package docker
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net"
"os"
"path/filepath"
"regexp"
@@ -29,6 +31,7 @@ import (
"github.com/docker/go-connections/nat"
controlapi "github.com/moby/buildkit/api/services/control"
buildkitclient "github.com/moby/buildkit/client"
"github.com/moby/buildkit/session"
"github.com/moby/buildkit/util/progress/progressui"
"go.uber.org/fx"
@@ -223,7 +226,9 @@ func buildEnvSlice(env map[string]string) []string {
return envSlice
}
// buildMounts converts volume mounts to Docker mount configuration.
// buildMounts converts volume mounts to Docker mount configuration. Docker,
// not upaas, creates a missing host path, because upaas runs in a container
// and cannot see the host's filesystem. An existing path is left alone.
func buildMounts(volumes []VolumeMount) []mount.Mount {
mounts := make([]mount.Mount, 0, len(volumes))
@@ -233,6 +238,9 @@ func buildMounts(volumes []VolumeMount) []mount.Mount {
Source: vol.HostPath,
Target: vol.ContainerPath,
ReadOnly: vol.ReadOnly,
BindOptions: &mount.BindOptions{
CreateMountpoint: true,
},
})
}
@@ -384,12 +392,17 @@ func (c *Client) ContainerLogs(
}
}()
logs, err := io.ReadAll(reader)
// A container without a terminal, as all of upaas's are, sends its
// output in frames, each with a header naming stdout or stderr. Both
// go to one buffer, in the order they were written.
var logs bytes.Buffer
_, err = stdcopy.StdCopy(&logs, &logs, reader)
if err != nil {
return "", fmt.Errorf("failed to read container logs: %w", err)
}
return string(logs), nil
return logs.String(), nil
}
// IsContainerRunning checks if a container is running.
@@ -659,11 +672,24 @@ func (c *Client) performBuild(
}
}()
buildSession, err := c.startBuildSession(ctx)
if err != nil {
return "", err
}
defer func() {
closeErr := buildSession.Close()
if closeErr != nil {
c.log.Error("failed to close build session", "error", closeErr)
}
}()
// Build with BuildKit: the stages of a multi-stage build are kept in
// its build cache, which Docker limits on its own, instead of being
// left behind as untagged images.
resp, err := c.docker.ImageBuild(ctx, tarArchive, dockertypes.ImageBuildOptions{
Version: dockertypes.BuilderBuildKit,
SessionID: buildSession.ID(),
Dockerfile: opts.DockerfilePath,
Tags: opts.Tags,
Remove: true,
@@ -699,6 +725,34 @@ func (c *Client) performBuild(
return "", nil
}
// startBuildSession attaches a BuildKit session to the daemon, as the docker
// command line does for a build. BuildKit asks the client, over the session,
// for registry access to fetch a base image that is not on the host; without
// a session, Docker Engine 27 fails the build with "no active sessions". The
// shared key is only used for a build context sent over the session; upaas
// sends the context with the build request. The caller closes the session.
func (c *Client) startBuildSession(ctx context.Context) (*session.Session, error) {
buildSession, err := session.NewSession(ctx, "")
if err != nil {
return nil, fmt.Errorf("failed to create build session: %w", err)
}
go func() {
runErr := buildSession.Run(ctx, func(
ctx context.Context,
proto string,
meta map[string][]string,
) (net.Conn, error) {
return c.docker.DialHijack(ctx, "/session", proto, meta)
})
if runErr != nil {
c.log.Error("build session failed", "error", runErr)
}
}()
return buildSession, nil
}
// scannerInitialBufferSize is the initial buffer size for the build log scanner.
const scannerInitialBufferSize = 64 * 1024 // 64KB
@@ -950,7 +1004,7 @@ func (c *Client) runGitClone(
return nil, fmt.Errorf("error waiting for git container: %w", err)
case status := <-statusCh:
// Always capture logs for the result
logs := c.gitContainerOutput(ctx, containerID)
logs, _ := c.ContainerLogs(ctx, containerID, "100")
if status.StatusCode != 0 {
return nil, fmt.Errorf(
@@ -978,31 +1032,6 @@ func (c *Client) runGitClone(
}
}
// gitContainerOutput returns the last 100 lines the git container wrote.
// Docker puts a header before each line a container without a terminal
// writes; stdcopy removes them so that the lines can be parsed.
func (c *Client) gitContainerOutput(
ctx context.Context,
containerID ContainerID,
) string {
reader, err := c.docker.ContainerLogs(ctx, containerID.String(), container.LogsOptions{
ShowStdout: true,
ShowStderr: true,
Tail: "100",
})
if err != nil {
return ""
}
defer func() { _ = reader.Close() }()
var output strings.Builder
_, _ = stdcopy.StdCopy(&output, &output, reader)
return output.String()
}
// Prefixes of the lines in the clone output that carry the commit checked
// out, in full and in git's short form.
const (
+39
View File
@@ -0,0 +1,39 @@
package docker //nolint:testpackage // tests unexported buildMounts
import (
"testing"
"github.com/docker/docker/api/types/mount"
"github.com/stretchr/testify/assert"
)
// TestBuildMountsCreatesMissingHostPath checks that every mount asks Docker
// to create its host path if it is missing, and keeps the rest of the volume
// as configured.
func TestBuildMountsCreatesMissingHostPath(t *testing.T) {
t.Parallel()
volumes := []VolumeMount{
{HostPath: "/srv/app/data", ContainerPath: "/data", ReadOnly: false},
{HostPath: "/srv/app/config", ContainerPath: "/etc/app", ReadOnly: true},
}
want := []mount.Mount{
{
Type: mount.TypeBind,
Source: "/srv/app/data",
Target: "/data",
ReadOnly: false,
BindOptions: &mount.BindOptions{CreateMountpoint: true},
},
{
Type: mount.TypeBind,
Source: "/srv/app/config",
Target: "/etc/app",
ReadOnly: true,
BindOptions: &mount.BindOptions{CreateMountpoint: true},
},
}
assert.Equal(t, want, buildMounts(volumes))
}
+161 -42
View File
@@ -262,48 +262,6 @@ func writeCloneOutput(w io.Writer) {
_, _ = stdout.Write([]byte("SHORT_SHA:1a2b3c4\n"))
}
// TestCloneRepoReadsCommit runs a clone against a fake Docker API and
// checks that the commit checked out is read from the clone's output, in
// full and in git's short form.
func TestCloneRepoReadsCommit(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/containers/create"):
_, _ = w.Write([]byte(`{"Id":"gitcontainer"}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w)
default:
_, _ = w.Write([]byte(`{}`))
}
},
))
t.Cleanup(srv.Close)
dockerAPI, err := client.NewClientWithOpts(
client.WithHost("tcp://" + srv.Listener.Addr().String()),
)
if err != nil {
t.Fatal(err)
}
c := &Client{docker: dockerAPI, log: slog.Default()}
result, err := c.CloneRepo(
t.Context(), "git@example.com:repo.git", mainBranch, "", "fake-key",
t.TempDir(), t.TempDir(),
)
if err != nil {
t.Fatal(err)
}
if result.CommitSHA != cloneCommit || result.ShortSHA != "1a2b3c4" {
t.Errorf("got commit %q, short %q", result.CommitSHA, result.ShortSHA)
}
}
// TestPerformBuildUsesBuildKit runs a build against a fake Docker API and
// checks that it asks for BuildKit and that BuildKit's progress reaches the
// build log as plain text.
@@ -333,6 +291,8 @@ func TestPerformBuildUsesBuildKit(t *testing.T) {
switch {
case strings.HasSuffix(r.URL.Path, "/version"):
_, _ = w.Write([]byte(`{"Version":"27.3.1","ApiVersion":"1.47"}`))
case strings.HasSuffix(r.URL.Path, "/session"):
serveSession(t, w, r, make(chan string, 1))
case strings.HasSuffix(r.URL.Path, "/build"):
if r.URL.Query().Get("version") != "2" {
http.Error(w, "not a BuildKit build", http.StatusBadRequest)
@@ -427,6 +387,8 @@ func TestPerformBuildFails(t *testing.T) {
case strings.HasSuffix(r.URL.Path, "/version"):
_, _ = fmt.Fprintf(w, `{"Version":%q,"ApiVersion":%q}`,
tt.engine, tt.apiVersion)
case strings.HasSuffix(r.URL.Path, "/session"):
serveSession(t, w, r, make(chan string, 1))
case strings.HasSuffix(r.URL.Path, "/build"):
_, _ = w.Write([]byte(tt.buildOutput))
default:
@@ -455,3 +417,160 @@ func TestPerformBuildFails(t *testing.T) {
})
}
}
// TestPerformBuildAttachesSession runs a build against a fake Docker API
// that, like the real daemon, fails the build with "no active sessions"
// unless the build names a session the client attached over the session
// endpoint. It also checks that the session is closed when the build ends.
func TestPerformBuildAttachesSession(t *testing.T) {
t.Parallel()
attached := make(chan string, 1)
sessionClosed := make(chan struct{})
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/version"):
_, _ = w.Write([]byte(`{"Version":"27.3.1","ApiVersion":"1.47"}`))
case strings.HasSuffix(r.URL.Path, "/session"):
serveSession(t, w, r, attached)
close(sessionClosed)
case strings.HasSuffix(r.URL.Path, "/build"):
id := r.URL.Query().Get("session")
attachedID := ""
// The daemon waits a few seconds for the build's session
// to attach.
if id != "" {
select {
case attachedID = <-attached:
case <-time.After(5 * time.Second):
}
}
if id == "" || attachedID != id {
_, _ = w.Write([]byte(`{"errorDetail":{"message":"no active sessions"},` +
`"error":"no active sessions"}`))
}
default:
t.Errorf("unexpected request to %s", r.URL.Path)
}
},
))
t.Cleanup(srv.Close)
dockerAPI, err := client.NewClientWithOpts(
client.WithHost("tcp://" + srv.Listener.Addr().String()),
)
if err != nil {
t.Fatal(err)
}
c := &Client{docker: dockerAPI, log: slog.Default()}
_, err = c.performBuild(t.Context(), BuildImageOptions{ContextDir: t.TempDir()})
if err != nil {
t.Fatal(err)
}
select {
case <-sessionClosed:
case <-time.After(5 * time.Second):
t.Error("the build's session was not closed when the build ended")
}
}
// serveSession answers a request to attach a session as the Docker daemon
// does: it switches the connection over to the session, sends the session's
// ID on attached, and holds the connection until the client closes it.
func serveSession(
t *testing.T,
w http.ResponseWriter,
r *http.Request,
attached chan<- string,
) {
t.Helper()
conn, _, err := http.NewResponseController(w).Hijack()
if err != nil {
t.Error(err)
return
}
defer func() { _ = conn.Close() }()
_, err = io.WriteString(conn, "HTTP/1.1 101 Switching Protocols\r\n"+
"Connection: Upgrade\r\nUpgrade: h2c\r\n\r\n")
if err != nil {
t.Error(err)
return
}
attached <- r.Header.Get("X-Docker-Expose-Session-Uuid")
_, _ = io.Copy(io.Discard, conn)
}
// TestPerformCloneReadsFramedLogs runs a clone against a fake Docker API that
// sends the clone container's output in frames, as Docker does for a
// container without a terminal, and checks that the output comes back as
// plain text and that the commit, in full and in git's short form, is read
// from it.
func TestPerformCloneReadsFramedLogs(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case strings.HasSuffix(r.URL.Path, "/containers/create"):
_, _ = w.Write([]byte(`{"Id":"gitcontainer"}`))
case strings.HasSuffix(r.URL.Path, "/wait"):
_, _ = w.Write([]byte(`{"StatusCode":0}`))
case strings.HasSuffix(r.URL.Path, "/logs"):
writeCloneOutput(w)
default:
_, _ = w.Write([]byte(`{}`))
}
},
))
t.Cleanup(srv.Close)
dockerAPI, err := client.NewClientWithOpts(
client.WithHost("tcp://" + srv.Listener.Addr().String()),
)
if err != nil {
t.Fatal(err)
}
c := &Client{docker: dockerAPI, log: slog.Default()}
dir := t.TempDir()
cfg := &cloneConfig{
repoURL: "git@example.com:repo.git",
branch: mainBranch,
sshPrivateKey: "fake-key",
containerDir: filepath.Join(dir, "repo"),
hostDir: filepath.Join(dir, "repo"),
keyFile: filepath.Join(dir, "deploy_key"),
hostKeyFile: filepath.Join(dir, "deploy_key"),
}
result, err := c.performClone(t.Context(), cfg)
if err != nil {
t.Fatal(err)
}
want := "Cloning into '/repo'...\nCOMMIT:" + cloneCommit + "\nSHORT_SHA:1a2b3c4\n"
if result.Output != want {
t.Errorf("got clone output %q, want %q", result.Output, want)
}
if result.CommitSHA != cloneCommit || result.ShortSHA != "1a2b3c4" {
t.Errorf("got commit %q, short %q", result.CommitSHA, result.ShortSHA)
}
}
+2
View File
@@ -21,6 +21,7 @@ import (
"sneak.berlin/go/upaas/internal/database"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/app"
"sneak.berlin/go/upaas/internal/service/deploy"
"sneak.berlin/go/upaas/templates"
)
@@ -194,6 +195,7 @@ func (h *Handlers) HandleAppDetail() http.HandlerFunc {
"EnvVars": envVars,
"Labels": labels,
"Volumes": volumes,
"NoVolumesWarning": deploy.NoVolumesWarning,
"Ports": ports,
"Deployments": deployments,
"LatestDeployment": latestDeployment,
+45
View File
@@ -0,0 +1,45 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/deploy"
)
// TestAppPageSaysFilesAreLostOnlyWhenAppHasNoVolumes checks that the app page
// shows deploy.NoVolumesWarning until the app gets a volume mount.
func TestAppPageSaysFilesAreLostOnlyWhenAppHasNoVolumes(t *testing.T) {
t.Parallel()
testCtx := setupTestHandlers(t)
createdApp := createTestApp(t, testCtx, "no-volumes-app")
renderAppPage := func() string {
request := httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/apps/"+createdApp.ID, nil,
)
request = addChiURLParams(request, map[string]string{"id": createdApp.ID})
recorder := httptest.NewRecorder()
testCtx.handlers.HandleAppDetail().ServeHTTP(recorder, request)
require.Equal(t, http.StatusOK, recorder.Code)
return recorder.Body.String()
}
assert.Contains(t, renderAppPage(), deploy.NoVolumesWarning)
volume := models.NewVolume(testCtx.database)
volume.AppID = createdApp.ID
volume.HostPath = "/srv/no-volumes-app"
volume.ContainerPath = "/data"
require.NoError(t, volume.Save(t.Context()))
assert.NotContains(t, renderAppPage(), deploy.NoVolumesWarning)
}
+11
View File
@@ -56,6 +56,12 @@ var (
ErrNoPreviousImage = errors.New("no previous image available for rollback")
)
// NoVolumesWarning is shown on the page of an app with no volume mounts and
// written into each of its deploy logs.
const NoVolumesWarning = "This app has no volume mounts, so the files it writes " +
"are lost whenever a deploy or rollback replaces its container; " +
"a restart of the container keeps them."
// logFlushInterval is how often to flush buffered logs to the database.
const logFlushInterval = time.Second
@@ -369,6 +375,11 @@ func (svc *Service) Deploy(
svc.logWebhookPayload(bgCtx, deployment, webhookEvent)
volumes, err := app.GetVolumes(bgCtx)
if err == nil && len(volumes) == 0 {
_ = deployment.AppendLog(bgCtx, NoVolumesWarning)
}
err = svc.updateAppStatusBuilding(bgCtx, app)
if err != nil {
return err
@@ -0,0 +1,79 @@
package deploy_test
import (
"context"
"log/slog"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/config"
"sneak.berlin/go/upaas/internal/database"
"sneak.berlin/go/upaas/internal/docker"
"sneak.berlin/go/upaas/internal/logger"
"sneak.berlin/go/upaas/internal/models"
"sneak.berlin/go/upaas/internal/service/deploy"
"sneak.berlin/go/upaas/internal/service/notify"
)
// deployLog deploys a new app, with one volume mount when withVolume is set,
// and returns the deploy's log. Docker is not connected, so the deploy fails
// at the git clone, after the start of its log is written.
func deployLog(t *testing.T, withVolume bool) string {
t.Helper()
log := logger.NewForTest(slog.New(slog.NewTextHandler(os.Stderr, nil)))
dataDir := t.TempDir()
cfg := &config.Config{DataDir: dataDir, HostDataDir: dataDir}
db := database.NewTestDatabase(t)
dockerClient, err := docker.New(nil, docker.Params{Logger: log, Config: cfg})
require.NoError(t, err)
notifySvc, err := notify.New(nil, notify.ServiceParams{Logger: log})
require.NoError(t, err)
svc, err := deploy.New(nil, deploy.ServiceParams{
Logger: log, Config: cfg, Database: db,
Docker: dockerClient, Notify: notifySvc,
})
require.NoError(t, err)
ctx := context.Background()
app := models.NewApp(db)
app.ID = "volumesapp-id"
app.Name = "volumesapp"
app.Branch = "main"
require.NoError(t, app.Save(ctx))
if withVolume {
volume := models.NewVolume(db)
volume.AppID = app.ID
volume.HostPath = "/srv/volumesapp"
volume.ContainerPath = "/data"
require.NoError(t, volume.Save(ctx))
}
err = svc.Deploy(ctx, app, nil, false)
require.ErrorIs(t, err, docker.ErrNotConnected)
deployments, err := app.GetDeployments(ctx, 1)
require.NoError(t, err)
require.Len(t, deployments, 1)
return deployments[0].Logs.String
}
func TestDeployLogSaysFilesAreLostOnlyWhenAppHasNoVolumes(t *testing.T) {
t.Parallel()
logWithoutVolumes := deployLog(t, false)
assert.Equal(t, 1, strings.Count(logWithoutVolumes, deploy.NoVolumesWarning),
logWithoutVolumes)
assert.NotContains(t, deployLog(t, true), deploy.NoVolumesWarning)
}
+2
View File
@@ -318,6 +318,8 @@
</tbody>
</table>
</div>
{{else}}
<p class="alert-warning">{{.NoVolumesWarning}}</p>
{{end}}
<form method="POST" action="/apps/{{.App.ID}}/volumes" class="flex flex-col sm:flex-row gap-2 items-end">
{{ .CSRFField }}