Logger writes to stdout, so any warning corrupts --json output #82
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found while implementing #64. Filing rather than fixing drive-by, since
it is outside that issue's scope and affects every
--jsoncommand.Problem
internal/log/log.go:73-78attaches both handlers toos.Stdout:Every
--jsonsubcommand writes its document to the same stream. Anylog record that is not suppressed therefore lands in the middle of the
document and makes it unparseable.
WARNandERRORare neversuppressed, so this is not hypothetical.
Reproduction
Any config file with permissions looser than 0600 triggers a
WARNatinternal/config/config.go:271during startup:vaultik snapshot list --json | jq .fails on that first line. The sameapplies to
snapshot verify --json,snapshot remove --json,prune --json, andremote info --json.Expected
Diagnostics go to stderr; stdout carries only the requested output. That
is the usual contract and the one a caller piping to
jqrelies on.Notes
internal/clialready passes aQuietflag through for some--jsonpaths, which suppressesINFObut notWARN/ERROR, soquiet mode is not a fix.
snapshot list --jsonworks around this locally as of #64: itsremote-listing warning is written to
v.Stderrdirectly rather thanthrough
logorv.UI. That workaround should be removed once thelogger itself writes to stderr.
destination, but it will move all
--verbose/--debugoutput tostderr too, which is a user-visible change worth confirming
deliberately.
Implementation plan
Implementing this together with
issue #97 on one branch,
since both are defects in
internal/logand #97 touches the samehandler construction site.
1. Move the logger to stderr
internal/log/log.goInitializebuilds both handlers overos.Stderrinstead ofos.Stdout, and the TTY/JSON handler choiceswitches to
term.IsTerminal(int(os.Stderr.Fd()))— the formatdecision has to follow the stream the records actually land on, not an
unrelated one.
AGENTS.mdpolicy 9 ("if stdout is not a terminal,output jsonl") is read as "if the log stream is not a terminal", which
is what it meant when logs went to stdout.
User-visible consequence, called out deliberately:
--verboseand--debugoutput moves to stderr as well.vaultik snapshot list -v > out.txtno longer captures the diagnostics. That is the intendedcontract and will be documented in
README.mdand in the PR body.--quiet/--cronsemantics are untouched: level selection inInitializeandui.Writerquiet handling are not modified, soUI.SetQuiet(true)still suppresses Begin/Complete/Info/Notice/Detail/Progress/Banner and still does not suppress Warning/Error (see
issue #84).
2. Remove the
snapshot_list.goworkaroundThe workaround goes away rather than staying. Concretely:
warnWhileListingis deleted. It hand-rolled structured-logformatting (
fmt.Fprintf(&line, " %v=%v", ...)plus akvPairSizeconstant) purely to reach a non-stdout writer; with the logger on
stderr,
log.Warnis the right call in both output modes.jsonOutputparameter threaded throughcollectRemoteSnapshotsanddescribeRemoteOnlySnapshotsonly everselected between those two writers, so it goes too.
reportJSONListingLimitsmoves fromfmt.Fprintf(v.Stderr, ...)tolog.Warnwith structured fields, for the same reason.warnRemoteListingFailedkeeps itsjsonOutputbranch, but for adifferent and still-true reason: table mode wants the prose
v.UI.Warningfline, andv.UIwrites to stdout, so JSON mode goesthrough
log.Warn. The comment is rewritten to say that instead ofciting this issue.
listingWarningcollect-then-emit machinery stays, with arewritten rationale. Its original justification (the chosen writer is
not concurrency-safe) is obsolete —
sloghandlers are safe forconcurrent use. What remains is that emitting from the worker
goroutines makes warning order depend on manifest-fetch completion
order; collecting and emitting in key order after
group.Wait()keeps diagnostics deterministic run to run. No comment is left
referencing this issue as pending work.
Existing tests that assert these warnings land on the injected
v.Stderrbuffer are updated to capture the process's real stderr,since that is where they now go. The assertions themselves (same
substrings, same guarantees) are preserved.
captureProcessStdout'sdoc comment, which currently cites this issue as the reason no
injectable sink exists, gets a matching
captureProcessStderrand anupdated explanation.
3. Verification
TestListSnapshots_JSONStdoutIsOnlyTheDocumentcontinues to pointevery stdout writer at one pipe and assert the capture parses as a
single JSON array.
mode 0644 (so
internal/config/config.gofires its insecure-permissions
WARN) plus afile://destination, thenvaultik snapshot list --json | jq .with stdout and stderr capturedseparately. stdout must parse; the warning must be on stderr.
make checkandscript/cibuild.Out of scope
The
--cronhelp string(issue #87) and the
suppression semantics themselves
(issue #84) are not
touched.
Implemented in
PR #107 (branch
fix-log-stdout), together withissue #97.
What changed
Initializenow builds both handlers overos.Stderr, and theTTY/JSON format choice tests
os.Stderrrather thanos.Stdout— theformat has to follow the stream the records land on, or a redirected
stderr gets colorized whenever stdout happens to be a terminal.
The
internal/vaultik/snapshot_list.goworkaround is removed ratherthan kept:
warnWhileListingdeleted, along with thekvPairSizeconstant andthe hand-rolled
%v=%vformatting it used to reach a non-stdoutwriter. Those warnings go through
log.Warnin every mode now.jsonOutputparameter is gone fromcollectRemoteSnapshotsanddescribeRemoteOnlySnapshots; it only ever selected between the twowriters.
reportJSONListingLimitsmoved fromfmt.Fprintf(v.Stderr, ...)tolog.Warnwith structured fields.warnRemoteListingFailedkeeps itsjsonOutputbranch, but thecomment now gives the reason that is still true: table mode wants the
prose
v.UI.Warningfline andv.UIwrites to stdout, so--jsonmode uses the logger.
listingWarningcollect-then-emit machinery stays, with arewritten rationale. Concurrency safety is no longer the reason —
sloghandlers are safe for concurrent use — but emitting from themanifest-fetch workers would order warnings by network timing, where
collecting and emitting in key order after
group.Wait()makes tworuns over the same damaged store produce the same diagnostics in the
same order.
No comment in the tree still refers to this issue as pending work. The
captureProcessStdouttest helper's doc, which cited it as the reasonno injectable sink exists, was rewritten too.
User-visible change
--verboseand--debugoutput moves to stderr as well, sovaultik snapshot list -v > out.txtno longer captures the diagnostics.README.mdgains a "stdout and stderr" section documenting the split,and the flag entries point at it. Called out prominently at the top of
the PR body.
Verification
Reproduced with the exact trigger from the report — a config file at
mode 0644 with a
file://destination — against a binary built frommain:Same config and command on this branch: stdout is
[]and nothing else,json.loadaccepts it, and theWARNis on stderr as a JSON record.--quietsemantics confirmed unchanged in the same run: it used-qand the warning still fired, per
issue #84. Level
selection and
ui.Writer's quiet handling were not touched, and--cronshares the same branch inInitialize.make checkgreen.script/cibuildexit 0 in 140s with the checklayers executed rather than replayed —
make fmt-check2.4s,make lint40.4s,make test58.5s under a freshCHECK_EPOCH— 14oklines and zero(cached)markers.One thing this does not finish
Filed as issue #106: the
startup banner is written to stdout by
internal/cli/entry.gobeforecobra parses, and
bannerSuppressedInArgsdoes not recognize--json.So
snapshot list --jsonstill emits two banner lines and a blank lineahead of the document unless
-qis passed. Different writer, differentpath, so it is filed rather than fixed here — but stdout is not fully
clean until it lands. The reproductions above use
-qfor that reason.