ctf — Collect To File#

Recursively collects source files into a single Markdown, JSON, JSONL or text
document, preserving paths relative to the source root. Built for assembling
LLM context, code-review bundles and auditable source snapshots.

Contents#

1. Why

2. Install

3. Quick start

4. Options reference

5. Output formats

6. Controlling what gets collected

7. Fitting a token budget

8. Self-update

9. Exit codes

10. Windows

11. Configuration file

12. Behaviour worth knowing

13. Development

14. Performance

15. License

1. Why#

Three situations come up constantly:

• LLM context. A model reasons better over one structured document than
  over twenty pasted fragments: it keeps the directory layout, the file
  boundaries and the language of each file. ctf produces exactly that, and
  can cap it at a token budget you specify.

• Cross-file review. A branch touching 20 files is easier to read as a
  single syntax-highlighted document than as 20 diffs.

• Snapshots and audits. A reproducible, timestamped aggregate of the
  sources, with per-file hashes on request.

ctf is a single self-contained script with no dependencies beyond a POSIX
userland. It does not shell out to eval, it does not execute anything it
reads, and it never writes outside the file you asked for.

2. Install#

git clone https://github.com/paulmann/ctf-collect-to-file.git
cd ctf-collect-to-file

# user-level install into ~/.local/bin
make install

# or system-wide
sudo make install PREFIX=/usr/local

# or without make
./install.sh --prefix "$HOME/.local"

Verify:

ctf --version      # ctf v4.1.0
ctf --print-config .

On Windows, use the PowerShell installer, which also puts the bare name ctf
on the PATH — see §10.1:

git clone https://github.com/paulmann/ctf-collect-to-file.git
cd ctf-collect-to-file
pwsh -NoLogo -NoProfile -NonInteractive -File .\install-ctf.ps1

Requirements: bash 4.2+ (CentOS/RHEL 7 baseline), plus awk, find,
sort, sed, head, tail, od, cat, wc, mktemp, stat.
Optional and auto-detected: file (faster binary check), git (--git),
sha256sum/shasum (--dedup, --metadata), curl/wget/python3/perl
(--update). Run ctf --print-config . to see what was detected.

3. Quick start#

# every PHP file under ./src into result.md
ctf php ./src result.md

# several extensions, modern option form
ctf -e php,js,ts -o bundle.md ./src

# everything the repository tracks or would track (honours .gitignore)
ctf --git all -o context.md .

# exactly 120k estimated tokens of Python, to the clipboard
ctf -e py --token-budget 120000 -o - . | pbcopy        # macOS
ctf -e py --token-budget 120000 -o - . | xclip         # Linux

# what would be collected, and how big would it be?
ctf --stats ./src
ctf --dry-run -e py ./src | head

4. Options reference#

ctf [OPTIONS] [EXTENSION] [SOURCE_DIR] [OUTPUT_FILE]

The three positional arguments are kept for compatibility with v1–v3 and are
equivalent to --ext, the source directory and --output. See
§12 for how positional arguments are resolved
when options are also present.

Collection#

Output#

Behaviour#

Self-update#

Environment#

5. Output formats#

Markdown (default)#

# Project Source Code Aggregate

| Field | Value |
|:------|:------|
| Generated (UTC) | 2026-10-08T12:00:00Z |
| Tool | ctf.sh v4.0.0 |
| Source | /repo |
| Extensions | php |
| Candidates | 42 |
| Collected | 37 |

---

### `src/Api/Client.php`

```php
<?php
...

Summary#

Skip reasons

The skip-reason table is the part that saves time: when a file you expected is
missing, the document itself tells you why.

### JSON / JSONL

`--format json` produces one object; `--format jsonl` produces one object per
line (streamable, `jq`-friendly):

```json
{"path":"src/Api/Client.php","lang":"php","bytes":4096,"lines":120,"sha256":"a1b2…","content":"<?php\n…"}

content is JSON-escaped; control characters other than \t, \n and \r
are removed, since JSON cannot carry them unescaped and source text never
contains them.

Plain text#

--format txt emits ===== path (lang, size) ===== separators and no Markdown
at all — convenient for diff, grep and diff-based review tools.

6. Controlling what gets collected#

Default excludes. VCS metadata (.git, .svn, .hg), dependency trees
(node_modules, vendor, .venv, __pycache__), build output (build,
dist, out, target, obj, release), caches, IDE directories and
generated files (*.min.js, *.map, lock files, *.pyc, shared objects,
executables). Excluded directories are pruned by find before descent, so
large node_modules trees cost nothing. List them with
--list-default-excludes, disable with --no-default-excludes.

The source root itself is never excluded. A tree rooted at ./dist,
./build or ./out is scanned normally — only directories below the root
are matched against the exclude list.

Patterns. Globs use bash pattern syntax, where * also crosses /:

ctf -E 'tests/*' -E '*.md' -E 'docs/**' -o ctx.md .
ctf -I 'src/**' -o src-only.md .
ctf --exclude-dir legacy,tmp -o clean.md .

Git-aware selection. --git all is usually what you want inside a
repository: it uses the index plus untracked-but-not-ignored files, so
.gitignore does the filtering for you and build output never appears.

Symlinks are skipped by default and reported as skip_symlink — never
silently. Pass -L to follow them.

7. Fitting a token budget#

The estimate is ceil(bytes / 4), which is conservative for source code.

ctf -e py --token-budget 120000 --stats .        # will it fit?
ctf -e py --token-budget 120000 -o ctx.md .      # default: truncate to fit
ctf -e py --token-budget 120000 --budget-action drop -o ctx.md .

Files are taken in --sort order until the budget is exhausted. With
--budget-action truncate (default) the file that does not fit whole is cut at
a line boundary and marked with an HTML comment; with drop it is skipped and
counted in budget_dropped. Both are reported in the summary.

Combine with --sort size to prioritise the largest files, or with
--max-size 64K to keep one generated monster from eating the window.

8. Self-update#

ctf --check-update          # local=4.1.0 remote=4.2.0 channel=main status=newer
ctf --update                # installs 4.2.0, keeps ctf.sh.bak-<timestamp>
ctf --update --update-channel latest

Safety properties, each covered by tests/test_update.sh:

• The download is validated before anything is overwritten: minimum size,
  a leading shebang, and a clean bash -n. An HTML error page, a truncated
  download or a syntactically broken script is refused and the installed copy
  is left untouched.

• A timestamped backup is created first; if the write fails, the previous
  version is restored.

• The new bytes are written through the existing inode, so ownership, mode
  and hard links survive.

• Sibling VERSION, ctf.ps1 and ctf.bat files are refreshed only if they
  already exist next to the script.

• Transport fallback: curl → wget → python3 → perl → bash /dev/tcp.
  The last one has no TLS, and says so instead of failing mysteriously.

• CTF_UPDATE_BASE_URL points the whole flow at a mirror (or a local test
  server), which is how the test suite exercises it without internet access.

9. Exit codes#

--check-update uses 3 for “a newer version exists”, so it can be used
directly as a CI gate:

ctf --check-update && echo "up to date" || echo "update available"

10. Windows#

ctf.ps1 is the Windows implementation; ctf.bat is the upstream launcher.
Both are CRLF in the repository — enforced by .gitattributes, because LF line
endings break labels and goto in cmd.exe.

ctf.bat php C:\projects\src C:\out\bundle.md
ctf.bat --check-update

.\ctf.ps1 -Ext php,js -Source .\src -Output bundle.md

The PowerShell port implements the collection, filtering, binary detection,
fence selection and Markdown/JSON output. Options that depend on POSIX
semantics (symlink modes, --files-from - from stdin) behave as documented in
ctf.ps1 -Help.

10.1 Install into PATH as ctf#

git clone https://github.com/paulmann/ctf-collect-to-file.git
cd ctf-collect-to-file
pwsh -NoLogo -NoProfile -NonInteractive -File .\install-ctf.ps1

Requirements: PowerShell 7.6+ (pwsh) for the installer. The engine itself
also runs on Windows PowerShell 5.1. The installer copies the engine byte for
byte, writes the launcher below, and appends one directory to the user PATH;
it never edits the engine and never touches the machine-wide PATH.

Installed layout:

<InstallRoot>\bin\ctf.cmd    launcher; only this directory goes on PATH
<InstallRoot>\lib\ctf.ps1    the engine, byte-identical to the repository
<InstallRoot>\lib\ctf.bat    upstream launcher, kept for parity
<InstallRoot>\install-ctf.ps1

After installation, ctf resolves to the same thing from both shells:

pwsh -NoProfile -Command "(Get-Command ctf).Source"   # ...\bin\ctf.cmd
cmd  /c "where ctf"                                    # ...\bin\ctf.cmd

10.2 Why the engine is not on PATH#

Putting ctf.ps1 in a PATH directory looks simpler and breaks in two
measured ways. Both were reproduced on PowerShell 7.6.

PowerShell prefers a .ps1, and does not bind --long-option arguments to
script parameters. A bare command name resolves to a .ps1 in preference to
a .cmd, and --name style arguments are not bound to parameters — the
literal string becomes a positional argument.

cmd.exe prefers a .bat, and the upstream ctf.bat enables delayed
expansion, which deletes ! characters from every argument. A path such as
B:\!1\src was silently truncated to B:, and the wrong directory was scanned
— with no error in the output at all.

The launcher written by the installer therefore sets
EnableExtensions DisableDelayedExpansion, calls the engine through
pwsh -File, and forwards both arguments and exit code. Paths containing !,
& or spaces survive intact; a path with ! is the regression this guards
against.

10.3 Option names differ between the ports#

Document layout and most options are the same, but not every name is. The
cause is a platform rule, not an oversight: PowerShell binds parameter names
case-insensitively, so bash option pairs that differ only in case — -e/-E
for --ext/--exclude, -v/-V for --verbose/--version — cannot both
have short forms. Verified by running the installed launcher.

Short forms that work on Windows: -e (--ext), -o (--output), -F
(--format), -d (--max-depth), -I (--include), -n (--dry-run),
-q (--quiet), -v (--verbose), -h (--help), -L (--follow). Use
the long form for everything else. ctf.ps1 -Help is the authoritative option
list for the Windows port.

--verbose is a silent no-op on Windows and is worth calling out, because
it fails without a word. The engine's own flag is named Trace and aliased to
v; -Help documents it as -Trace — "Debug output (equivalent of the bash
--verbose)". The literal argument --verbose matches no parameter of the
script, but PowerShell binds it to its common -Verbose parameter, so the
call succeeds, prints nothing extra, and exits 0. Measured through the
installed launcher on a tree containing a binary file — the one code path that
emits a debug line:

Use -v (or -Trace) when you need the skip reasons. Note that the port emits
debug output from a single call site — the deterministic binary-skip during
collection — so -v is far quieter than the bash --verbose, which reports
every skip with its reason. For a per-file inventory of what was selected and
skipped, use --dry-run (one TSV row per file), or read the skip-reasons table
at the end of the produced document.

Exit codes. A PowerShell parameter-binding failure exits 1, not 2; code
2 comes from the engine's own usage checks. --check-update exits 3 when a
newer version exists, so it still works as a CI gate.

10.4 Line endings and linting#

.bat and .ps1 are CRLF; .sh, Makefile and .gitattributes are LF. CI
enforces this, together with a parser check of ctf.ps1, so normalising a
Windows file to LF fails the build. ctf.ps1 deliberately has no #Requires
directives, so that the port stays loadable on Windows PowerShell 5.1 — do not
add them without a matching decision.

11. Configuration file#

# ~/.ctfrc — plain VAR=VALUE, no code
CTF_FORMAT=md
CTF_SORT=path
CTF_TOKEN_BUDGET=120000
CTF_DEFAULT_EXCLUDES=1
CTF_TITLE="Project sources"

ctf -c ~/.ctfrc -e py .
CTF_CONFIG=~/.ctfrc ctf -e py .
ctf --print-config .        # what is actually in effect, and why

The file is treated strictly as data:

• only variables on the CTF_* whitelist are accepted; anything else is
  ignored with a warning;

• values are never passed to eval — there is no eval in the bash code;

• values containing $(, backticks, ;, |, &, <, > or $ are
  rejected;

• a group- or world-writable config is ignored with a warning, because
  such a file could be planted by another local user.

12. Behaviour worth knowing#

Positional arguments fill the slots EXTENSION, SOURCE_DIR,
OUTPUT_FILE in that order, but only slots that no option has already filled,
and an argument naming an existing directory is always taken as SOURCE_DIR:

Binary detection uses two signals, both false-positive-free for real text:
a NUL byte in the first 8 KiB, or more than 5 % C0 control bytes (excluding
\t \n \v \f \r). High bytes are not evidence of binary — otherwise every
non-English source file would be dropped. When file(1) is present its verdict
is used first. Known limitation: a file under ~1 KiB of high-entropy data with
no NUL and very few control bytes can pass as text; use --ext or
--binary always when that matters.

Fences are computed per file. A fenced block is closed only by a line
consisting of nothing but fence characters (optionally indented up to 3
spaces), so ctf finds the longest such run in the file and emits a fence one
character longer — choosing backticks or tildes, whichever is shorter. Content
is therefore never able to break out of its block.

A file without a trailing newline still gets its closing fence on a line of
its own.

File names may contain any byte except / and NUL, and ctf handles
spaces, quotes, backticks, $, ;, tabs and newlines — verified by
tests/test_safety.sh, which also asserts that no payload inside a file name
is ever executed. Names containing CR/LF/TAB are flattened for display only
(headings, TOC, TSV); content is copied byte-for-byte.

Reproducibility. With --no-timestamp (or SOURCE_DATE_EPOCH), identical
input yields byte-identical output, so a snapshot can be committed and diffed.

The output file is never collected into itself, even when it lives inside
the scanned tree; the skip is reported as skip_output_file.

Atomic writes. The document is assembled in temporary files next to the
destination and moved into place; a symlinked destination is written through,
so the link is not replaced by a regular file. Temporary files are removed by
an EXIT trap.

13. Development#

make test          # full suite: 8 files, 243 assertions, no network, no root
make lint          # shellcheck --severity=style on ctf.sh and tests
make check         # lint + test
make install       # install ctf.sh, ctf.ps1, ctf.bat
make uninstall

Or directly:

tests/run_tests.sh                 # everything
tests/run_tests.sh --filter binary # one file
tests/run_tests.sh --verbose       # print every passing assertion
tests/run_tests.sh --keep-tmp      # keep the scratch directory
CTF=/path/to/other/ctf.sh tests/run_tests.sh   # test a different copy

Layout:

ctf.sh             the tool (bash 4.2+, 2222 lines, shellcheck-clean at style level)
ctf.ps1            Windows implementation (PowerShell 5.1+, 1139 lines)
ctf.bat            Windows launcher (68 lines, CRLF)
VERSION            single source of truth for the version, read by --update
tests/             8 test files, 243 assertions; sources ctf.sh, no framework
  run_tests.sh     runner (--filter, --verbose, --keep-tmp)
  lib.sh           assertions, fixtures, ctf_run/ctf_stats helpers
  test_cli.sh      options, positional forms, exit codes
  test_collect.sh  extensions, excludes, symlinks, depth, size, git, --files-from
  test_binary.sh   binary detection: determinism and false positives
  test_output.sh   document structure, fences, formats, TOC, budget, BOM
  test_safety.sh   atomic writes, hostile file names, config file, temp hygiene
  test_update.sh   self-update end-to-end against a local HTTP server
  test_parity.sh   ctf.sh vs ctf.ps1: byte-identical output and CLI parity
  test_meta.sh     versions, shellcheck, bash 4.2 portability, docs consistency
docs/ANALYSIS.md   audit of v3.1.0: 20 defects with reproduction commands
docs/DECISIONS.md  20 design decisions: context, choice, cost
install.sh         POSIX installer (no make required)
Makefile           install / uninstall / test / lint / check / dist
install-ctf.ps1    Windows installer: copies the engine, writes the launcher,
                   adds its directory to the user PATH (see §10.1)
SKILL.md           agent-facing usage and installation guide
.github/workflows/ci.yml

CI runs shellcheck, bash -n, the suite (Ubuntu and macOS), version
consistency across VERSION/ctf.sh/README/CHANGELOG, a PowerShell parse
check of ctf.ps1, and line-ending policy (.bat/.ps1 CRLF, .sh LF).
When pwsh is present — it is on ubuntu-latest — test_parity.sh also
compares the two implementations byte-for-byte.

Linting policy (see docs/DECISIONS.md, DR-18): ctf.sh and install.sh
must be clean at --severity=style; tests must be clean at
--severity=warning; deliberate suppressions carry an explanatory directive
instead of a global .shellcheckrc.

14. Performance#

Metadata reads are batched — sizes via stat --printf, binary classification
via grep -IlZ, and lines/fence-runs/byte-sums via one awk pass per ~200
files. The only per-file process left is the cat that copies content, and
hot-path helpers return through globals instead of command substitution, so no
subshell is forked per file.

Measured in a container, 2 CPU:

Measured in a container with 2 CPU. Batched output is byte-identical to the
unbatched implementation, which the determinism tests assert.

Batched output is byte-identical to the unbatched implementation (asserted by
the determinism tests).

15. License#

PolyForm Noncommercial 1.0.0 — see LICENSE.

Author: Mikhail Deynekin — Mikhail@Deynekin.com — https://deynekin.com