From e6c901d2e5430a3815109b38ced11a4f695f0226 Mon Sep 17 00:00:00 2001 From: Crash Over Burn Date: Tue, 14 Apr 2026 13:05:33 +0000 Subject: Port upstream patches: Cmdliner 2.x, lockfile auto-creation, schema upgrade, Fossil VCS Ported from upstream darcs repository (v1.1.2): - Cmdliner 2.x compatibility fixes (variable shadowing) - Lockfile auto-creation when missing - Schema upgrade command with backup/rollback - Fossil VCS support (new VCS type) - Clean up Cmdliner warning for unescaped $PWD Files modified: - lib/schema.ml (new): Schema versioning module - lib/nixtamal.ml: Add upgrade function, Fossil meld support - lib/error.ml: Add Fossil to prefetch_method, Upgrade error - lib/input.ml: Add Fossil module, Kind variant - lib/prefetch.ml: Add Fossil prefetch with SRI hash support - lib/manifest.ml: Add Fossil codec - lib/lockfile.ml: Add Fossil lockfile type - lib/lock_loader.ml: Add Fossil feature flag - lib/input_foreman.ml: Add Fossil display and prefetch check - bin/cmd.ml: Cmdliner 2.x fixes, add Upgrade command - bin/dune, lib/dune, test/dune: Deprecation flags Builds successfully with all tests passing. --- .gitignore | 2 + AGENTS.md | 72 +++----- README.asciidoc | 39 ++++- README.rst | 269 ----------------------------- bin/cmd.ml | 84 ++++++++- bin/dune | 1 + bin/main.ml | 1 + flake.lock | 8 +- flake.nix | 15 +- lib/dune | 2 + lib/error.ml | 10 +- lib/input.ml | 97 ++++++++++- lib/input_foreman.ml | 44 +++++ lib/lock_loader.ml | 6 + lib/lockfile.ml | 83 +++++++++ lib/manifest.ml | 173 ++++++++++++++++++- lib/nixtamal.ml | 161 +++++++++++++++++ lib/prefetch.ml | 119 +++++++++++++ lib/schema.ml | 31 ++++ llm/2026-01-07-Session-Changes-Summary.md | 33 ---- llm/2026-02-07-attribution-audit.md | 25 --- llm/2026-02-07-phase1-flake-integration.md | 133 -------------- llm/README.md | 41 ----- nixtamal.opam | 2 +- test/dune | 1 + 25 files changed, 871 insertions(+), 581 deletions(-) create mode 100644 .gitignore delete mode 100644 README.rst create mode 100644 lib/schema.ml delete mode 100644 llm/2026-01-07-Session-Changes-Summary.md delete mode 100644 llm/2026-02-07-attribution-audit.md delete mode 100644 llm/2026-02-07-phase1-flake-integration.md delete mode 100644 llm/README.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2e975c9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +result +llm/ diff --git a/AGENTS.md b/AGENTS.md index ef06157..6e2a2f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,24 +1,21 @@ # AGENTS.md for nixtamal project -## Project Overview -**Nixtamal** is a Nix version pinning tool designed as an alternative/complement to flakes. It provides sophisticated dependency management with first-class support for Darcs, Pijul, and other VCS systems that flakes don't handle well. +## Documentation -**Strategic Position**: While nixtamal is philosophically an alternative to flakes, toastal is implementing dual support to provide ecosystem bridge capabilities while maintaining the project's core mission. +User-facing documentation is in `README.asciidoc`. Agent notes go in `./llm/`. ## Build/Lint/Test Commands ### Traditional Nix Commands -- Build: `dune build` -- Test all: `dune runtest` +- Build: `nix-build release.nix` or `nix build` +- Test all: `nix-build release.nix -A check` or `dune runtest` - Run single test: `dune runtest --filter ` (use with alcotest test names) - Format: `dune build @fmt` (ocamlformat) - Clean: `dune clean` -### Nix Integration Commands -- Traditional build: `nix-build` or `nix-build release.nix` -- Development shell: `nix-shell` or `nix-shell shell.nix` -- Flake build (Phase 1+): `nix flake build` or `nix build .#nixtamal` -- Flake shell (Phase 1+): `nix develop` or `nix shell .#devShell` +### Nix Flake Commands +- Build: `nix flake build` or `nix build .#nixtamal` +- Dev shell: `nix develop` or `nix shell .#devShell` ## Code Style Guidelines - **Formatting**: Use ocamlformat (auto-formatted via dune @fmt) @@ -31,47 +28,30 @@ - **Line Length**: No strict limit, but aim for readability - **Pattern Matching**: Exhaustive, use `|` for clarity -## Phase 1: Flake Integration Planning +## Ongoing Integrations -### Week 1 Objectives -1. **Create `flake.nix`** with core outputs wrapping existing `release.nix` -2. **Preserve existing infrastructure** as primary build method -3. **Implement basic flake outputs**: - - `packages.${system}.default` (nixtamal package) - - `packages.${system}.nixtamal` (explicit package name) - - `devShells.${system}.default` (development environment) - - `checks.${system}` (test suite integration) - - `lib` outputs for ecosystem integration +### Nilla Framework (In Progress) -### Week 1 Deliverables -- `flake.nix` with minimal working flake interface -- `flake.lock` generation workflow -- Documentation updates explaining dual approach -- Testing both traditional and flake workflows +Adding first-class support for https://github.com/nilla-nix/nilla[Nilla], a Nix framework with loaders and module system. -### Strategic Constraints -- **Maintain philosophical consistency**: Flakes as interface, not replacement -- **Preserve existing workflows**: No breaking changes to current Nix infrastructure -- **Complement over compete**: Position as bridge between traditional Nix and flake ecosystems +**Status**: Types, codecs, and manifest serialization implemented. Prefetch not yet implemented. + +**Files modified**: +- `lib/input.ml` - Nilla module with Reference type, Kind variant +- `lib/manifest.ml` - KDL codec for Nilla input type +- `lib/error.ml` - Added Nilla to prefetch_method, Not_implemented error +- `lib/prefetch.ml` - Stub returning "not yet implemented" +- `lib/input_foreman.ml` - Display and needs_prefetch logic +- `lib/lock_loader.ml` - Feature constant +- `lib/lockfile.ml` - Lockfile serialization +- `lib/nixtamal.ml` - meld_input_with_lock support + +### Future Integrations +- Additional VCS support as needed +- Migration utilities (flakes, npins, niv) ## Agent Notes - After each major change, create comprehensive notes in `./llm/` folder -- Use the template in `./llm/README.md` +- Use the template in `./llm/README.md` (if present) - Document learnings, challenges, solutions, and insights for future reference - **Flake integration changes** should be specifically documented with strategic reasoning - -## Flake Integration Architecture - -### Design Principles -1. **Wrapper Pattern**: flake.nix imports and wraps existing release.nix infrastructure -2. **Dual Interface**: Both traditional Nix and flakes remain fully functional -3. **Ecosystem Bridge**: Expose nixtamal capabilities to flake users without compromising core mission -4. **Incremental Adoption**: Users can gradually engage with flakes without abandoning nixtamal - -### Key Architectural Decisions -- Keep `nix/tamal/` system as primary input management -- Use flakes only for build/packaging interface layer -- Maintain backward compatibility with existing nixtamal projects -- Provide migration utilities in future phases - -No Cursor or Copilot rules found. \ No newline at end of file diff --git a/README.asciidoc b/README.asciidoc index b7ebc25..607f02e 100644 --- a/README.asciidoc +++ b/README.asciidoc @@ -6,7 +6,7 @@ :author: toastal == Pronunciation -/nɪʃ.təˈmal/ _or_ /ˈnɪkstəˌmɑːl/ +/nɪks.təˈmal/ ("nix-tamal") Maker:: toastal @@ -18,7 +18,11 @@ Languages:: * KDL * JSON -Version:: 0.0.9-alpha (alpha stage) +Version:: 1.1-alpha (alpha stage) + +**Community-maintained fork**: Addressing the original project's lack of reasonable maintenance, this fork prioritizes long-term maintainability, compatibility, and active development. + +*Original project:* link:https://darcs.toastal.in.th/nixtamal/trunk/[darcs.toastal.in.th/nixtamal/trunk] == Purpose @@ -48,6 +52,33 @@ Future goals (planned but not yet implemented): WARNING: As this software is in the alpha stage, the maker reserves the right make breaking changes file schemas & CLI API. Additionally, anything after tagged, the maker reserves the right to obliterate & amend patches. +== Ongoing Integrations + +Nixtamal is being extended to support additional frameworks and tools: + +=== Nilla Framework + +https://github.com/nilla-nix/nilla[Nilla] is a Nix framework with loaders and module system. Nixtamal provides first-class support for Nilla inputs, including manifest serialization, lockfile integration, and prefetch functionality. + +[source,kdl] +---- +my-input { + nilla { + repository "https://github.com/user/nilla-project.git" + branch "main" + path "./nilla.nix" + } +} +---- + +Attributes: +* `repository` - Git repository URL (templated) +* `mirrors` - Repository mirrors (optional, templated) +* `branch` or `ref` - Git reference point +* `path` - Path to nilla.nix (defaults to `./nilla.nix`) + +Nixtamal prefetch validates that the nilla.nix file exists at the specified path within the fetched repository. The implementation uses standard nix prefetch tooling for consistency and reliability. + == Quickstart === Set up @@ -178,9 +209,9 @@ $ nix-build release.nix $ nix-build release.nix -A nixtamal ---- -=== Building with Flakes (New: Phase 1 Integration) +=== Building with Flakes (New: Phase 1 Integration with flake-parts) -Nixtamal now supports dual workflow compatibility, providing both traditional Nix and modern flake interfaces. The flake integration acts as an ecosystem bridge while maintaining nixtamal's core philosophy as a flake alternative/complement. +Flakes use Determinate Nix and nixpkgs (github:DeterminateSystems/nix, NixOS/nixpkgs/nixos-unstable-small) for strong community support for Nix creator Eelco Dolstra's vision. Basic flake build diff --git a/README.rst b/README.rst deleted file mode 100644 index 6c599bd..0000000 --- a/README.rst +++ /dev/null @@ -1,269 +0,0 @@ -.. - ┏┓╻+╻ ╱┏┳┓┏┓┏┳┓┏┓╻ - ┃┃┃┃┗━┓╹┃╹┣┫┃┃┃┣┫┃ - ╹┗┛╹╱ ╹ ╹ ╹╹╹ ╹╹╹┗┛ -================================================================================ -Nixtamal -================================================================================ --------------------------------------------------------------------------------- -Fulfilling input pinning for Nix (& hopefully more) --------------------------------------------------------------------------------- - -:author: toastal - -.. role:: ac -.. role:: del -.. role:: t - -Pronunciation - /nɪʃ.təˈmal/ *or* /ˈnɪkstəˌmɑːl/ -Alternative spellings - • ·𐑯𐑦𐑖𐑑𐑩𐑥𐑭𐑤 -Maker - toastal -Put out - 2025-12 -Languages - • OCaml - • Nix - • :ac:`KDL` - • :ac:`JSON` - - -Purpose -================================================================================ - -Nixtamal is a tool to pin input version like its predecessors :t:`niv`, -:t:`npins`, :t:`Pinch`, :t:`Yae`\, & so on — as well as Nix’s experimental -:t:`flakes`. Features include: - -• keeps a stable reference pin to supported :ac:`VCS`\s -• supports mirrors for fetching [1]_ -• supports patch-based :ac:`VCS`\s, like Pijul & Darcs, in a first-class sense - (tho ``nixpkgs`` will be required due to Nix ``builtins`` fetchers - limitations) -• uses a :ac:`KDL` manifest file with templating instead of :ac:`CLI` input -• supports arbitrary commands for getting the latest change for inputs -• refreshes inputs; skips frozen -• locks new sources -• supports arbitrary commands for getting the latest change which can be used to - avoid costly downloads/clones as well as for templating for inputs -• does not give any special privilege to any specific code forges -• source code & community will never be hosted on a proprietary, - privacy-invasive, megacorporate platform with obligations to shareholders or - venture capital -• licensed for your freedom -• ML-family programming (feels closer to Nix) - -Future goals: - -• migrations from prior manifest × lockfile versions -• migrations from Flakes, Npins, & Niv -• more :ac:`VCS`\s -• ``nixtamal heal`` for common pitfalls in ``manifest.kdl`` -• :ac:`TUI`? -• provide a flake-like specification for project layout, but with less holes - -.. warning:: - - As this software is in the alpha stage, the maker reserves the right make - breaking changes file schemas & :ac:`CLI` :ac:`API`. Additionally, anything - after tagged, the maker reserves the right to obliterate & amend patches. - -Quickstart -================================================================================ - -Set up --------------------------------------------------------------------------------- - -.. code:: console - - $ nixtamal set-up - ┏┓╻+╻ ╱┏┳┓┏┓┏┳┓┏┓╻ - ┃┃┃┃┗━┓╹┃╹┣┫┃┃┃┣┫┃ - ╹┗┛╹╱ ╹ ╹ ╹╹╹ ╹╹╹┗┛ - - Creating Nixtamal directory @ ./nix/tamal - Writing new Nixtamal EditorConfig @ ./nix/tamal/.editorconfig … - Fetching latest value for 「nixpkgs」 … - Prefetching input 「nixpkgs」 … (this may take a while) - Prefetched 「nixpkgs」. - Making manifest file @ version:0.1.1 - - $ tree nix/tamal - nix/tamal - ├── default.nix - ├── lock.json - └── manifest.kdl - - 1 directory, 3 files - - -Use with a Nix project — such as in a ``release.nix`` --------------------------------------------------------------------------------- - -.. code:: nix - - let - inputs = import nix/tamal { }; - pkgs = import inputs.nixpkgs { }; - in - { - inherit (pkgs) hello; - } - -Add a new input to pin --------------------------------------------------------------------------------- - -See: ``_ - -.. code:: console - - $ nixtamal tweak - -Opens text editor. & After editing … - -Lock or refresh you inputs --------------------------------------------------------------------------------- - -.. code:: console - - $ nixtamal lock - $ nixtamal refresh - -What next? --------------------------------------------------------------------------------- - -As they say: read the manpages - -.. code:: console - - $ man nixtamal - $ man nixtamal-manifest - - -Building / hacking -================================================================================ - -Working setup --------------------------------------------------------------------------------- - -If you don’t have Darcs install, you can use from Nixpkgs such as - -.. code:: console - - $ nix-shell -p darcs - -After/else - -.. code:: console - - $ darcs clone https://darcs.toastal.in.th/nixtamal/trunk/ nixtamal - $ darcs setpref boringfile .boring - $ cd nixtamal - -Development environment setup --------------------------------------------------------------------------------- - -Base Nix shell - -.. code:: console - - $ nix-shell - -Or with Direnv - -.. code:: console - - $ echo "use nix" >> .envrc - $ direnv allow - -Building with Dune --------------------------------------------------------------------------------- - -.. code:: console - - $ dune build - -Building with Nix --------------------------------------------------------------------------------- - -Basic - -.. code:: console - - $ nix-build - -Everything else - -.. code:: console - - $ nix-build release.nix - $ nix-build release.nix -A nixtamal - -Darcs hooks (can skip) --------------------------------------------------------------------------------- - -.. code:: console - - $ $EDITOR _darcs/prefs/defaults - -.. code:: - - apply posthook nix-shell --run mk-darcs-weak-hash && nix-build --no-out-link release.nix - obliterate posthook nix-shell --run mk-darcs-weak-hash - record posthook nix-shell --run mk-darcs-weak-hash - -Hooks here can: - • Build the entire project before applying patches to make sure it works. - • Show the WeakHash ``_darcs/weak_hash`` which is good for querying project - state, such as for ``latest-cmd``s (hint, hink). - - -License -================================================================================ - -Depending on the content, this project is licensed under one of - -• :t:`GNU General Public License, version 3.0 later` (``GPL-3.0-or-later``) -• :t:`GNU Lesser General Public License version 2.1 or later` - (``LGPL-2.1-or-later``) with & without the :t:`OCaml LGPL Linking Exception` - (``OCaml-LGPL-linking-exception``) -• :t:`ISC License` (``ISC``) -• :t:`Creative Commons Attribution-ShareAlike 4.0 International` - (``CC-BY-SA-4.0``) -• :t:`Mozilla Public License Version 2.0` (``MPL-2.0``) - -For details read ``LICENSE.txt`` with full license texts at ``license/``. - - -Pitching in -================================================================================ - -Currently this is best done by sending a patchset to -`toastal+nixtamal@posteo.net`_ or :ac:`DM` me a remote to clone @ -`toastal@toastal.in.th`_. - -Community is in an :ac:`XMPP` :ac:`MUC` (chatroom) with future hopes to have an -:ac:`IRC` gateway. Join @ . - -.. - Additionally, please read the ``PITCHING_IN.rst`` file for other - information/expectations. - -.. _toastal+nixtamal@posteo.net: mailto:toastal+nixtamal@posteo.net -.. _toastal@toastal.in.th: xmpp:toastal@toastal.in.th - - -Funding -================================================================================ - -See choices at the `maker’s website `_. - - -.. [1] :ac:`WIP` with upstream Nixpkgs - - • :del:`Darcs: https://github.com/NixOS/nixpkgs/pull/467172` - • :del:`Pijul: https://github.com/NixOS/nixpkgs/pull/467890` - -.. vim: set textwidth=80 diff --git a/bin/cmd.ml b/bin/cmd.ml index a411f43..533584d 100644 --- a/bin/cmd.ml +++ b/bin/cmd.ml @@ -29,7 +29,7 @@ module Global = struct & info ["directory"] ~env - ~doc: "Working directory for Nixtamal-related files (default: $PWD/nix/tamal)" + ~doc: "Working directory for Nixtamal-related files (default: \\$PWD/nix/tamal)" ~docv: "PATH" ) @@ -186,8 +186,9 @@ module Set_up = struct $ nixpkgs_revision_arg ) in + let eio_env = env in Term.( - const (fun glb -> Global.run ~env glb @@ run) + const (fun glb -> Global.run ~env: eio_env glb @@ run) $ Global.args $ nixpkgs_arg ) @@ -210,8 +211,9 @@ module Check_soundness = struct let term ~env = let open Cmdliner in + let eio_env = env in Term.( - const (fun glb -> Global.run ~env glb @@ run) + const (fun glb -> Global.run ~env: eio_env glb @@ run) $ Global.args ) @@ -233,8 +235,9 @@ module Tweak = struct let term ~env = let open Cmdliner in + let eio_env = env in Term.( - const (fun glb -> Global.run ~env glb @@ run) + const (fun glb -> Global.run ~env: eio_env glb @@ run) $ Global.args ) @@ -255,8 +258,9 @@ module Show = struct let term ~env = let open Cmdliner in + let eio_env = env in Term.( - const (fun glb -> Global.run ~env glb @@ run) + const (fun glb -> Global.run ~env: eio_env glb @@ run) $ Global.args ) @@ -291,8 +295,9 @@ module Lock = struct & info [] ~docv: "INPUT_NAME" ~doc: "Input names to lock (if already locked, will skip)." ) in + let eio_env = env in Term.( - const (fun glb force -> Global.run ~env glb @@ run force) + const (fun glb force -> Global.run ~env: eio_env glb @@ run force) $ Global.args $ force_arg $ names_arg @@ -315,8 +320,9 @@ module List_stale = struct let term ~env = let open Cmdliner in + let eio_env = env in Term.( - const (fun glb -> Global.run ~env glb @@ run) + const (fun glb -> Global.run ~env: eio_env glb @@ run) $ Global.args ) @@ -345,11 +351,73 @@ module Refresh = struct & info [] ~docv: "INPUT_NAME" ~doc: "Input names to refresh." ) in + let eio_env = env in Term.( - const (fun glb -> Global.run ~env glb @@ run) + const (fun glb -> Global.run ~env: eio_env glb @@ run) $ Global.args $ names_arg ) let cmd ~env = Cmdliner.Cmd.v info (term ~env) end + +module Upgrade = struct + let info = + Cmdliner.Cmd.info + "upgrade" + ~doc: "Upgrade lockfile & manifest to current schema version." + ~exits: Cmdliner.Cmd.Exit.defaults + ~man: common_man + + let run ~env: _ ~domain_count: _ from to_ dry_run : unit = + let open Nixtamal in + match upgrade ?from ?to_ ~dry_run () with + | Ok () -> () + | Error err -> failwith (Fmt.str "%a" Nixtamal.Error.pp err) + + let term ~env = + let eio_env = env in + let open Cmdliner in + let version_conv : Nixtamal.Schema.Version.t Arg.conv = + Arg.conv + ~docv: "VERSION" + ((fun s -> + match Nixtamal.Schema.Version.of_string s with + | Some v -> Ok v + | None -> + let valid = Fmt.(array ~sep: comma Nixtamal.Schema.Version.pp) in + Error (`Msg (Fmt.str "Invalid version %s. Valid versions: %a" s valid Nixtamal.Schema.Version.versions)) + ), + Nixtamal.Schema.Version.pp) + in + let from_arg = + let doc = "Upgrade from a specific version. Must match the lockfile version." in + Arg.( + value + & opt (some version_conv) None + & info ["from"] ~doc ~docv: "VERSION" + ) + and to_arg = + let doc = "Upgrade to a specific version. Must be newer than --from." in + Arg.( + value + & opt (some version_conv) None + & info ["to"] ~doc ~docv: "VERSION" + ) + and dry_run_arg = + Arg.( + value + & flag + & info ["dry-run"] ~doc: "Show what would be upgraded without making changes." + ) + in + Term.( + const (fun glb -> Global.run ~env: eio_env glb @@ run) + $ Global.args + $ from_arg + $ to_arg + $ dry_run_arg + ) + + let cmd ~env = Cmdliner.Cmd.v info (term ~env) +end diff --git a/bin/dune b/bin/dune index 5705f13..f36fcea 100644 --- a/bin/dune +++ b/bin/dune @@ -13,6 +13,7 @@ logs.cli logs.fmt uri) + (flags (:standard -alert -deprecated)) (preprocess (pps ppx_deriving.show ppx_deriving.eq ppx_deriving.ord ppx_deriving.make))) diff --git a/bin/main.ml b/bin/main.ml index bdb25e7..739776f 100644 --- a/bin/main.ml +++ b/bin/main.ml @@ -27,6 +27,7 @@ let cmd ~env = Cmd.Lock.cmd ~env; Cmd.List_stale.cmd ~env; Cmd.Refresh.cmd ~env; + Cmd.Upgrade.cmd ~env; ] in Cmdliner.Cmd.group info subcommands diff --git a/flake.lock b/flake.lock index 57a7d06..a7beb44 100644 --- a/flake.lock +++ b/flake.lock @@ -20,17 +20,17 @@ }, "nixpkgs": { "locked": { - "lastModified": 1770380644, - "narHash": "sha256-P7dWMHRUWG5m4G+06jDyThXO7kwSk46C1kgjEWcybkE=", + "lastModified": 1767364772, + "narHash": "sha256-fFUnEYMla8b7UKjijLnMe+oVFOz6HjijGGNS1l7dYaQ=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "ae67888ff7ef9dff69b3cf0cc0fbfbcd3a722abe", + "rev": "16c7794d0a28b5a37904d55bcca36003b9109aaa", "type": "github" }, "original": { "owner": "NixOS", - "ref": "nixpkgs-unstable", "repo": "nixpkgs", + "rev": "16c7794d0a28b5a37904d55bcca36003b9109aaa", "type": "github" } }, diff --git a/flake.nix b/flake.nix index 3d8d9a5..115a9d1 100644 --- a/flake.nix +++ b/flake.nix @@ -6,19 +6,24 @@ description = "Nixtamal - A Nix version pinning tool with first-class support for Darcs, Pijul, and other VCS systems"; inputs = { - nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; + nixpkgs.url = "github:NixOS/nixpkgs/16c7794d0a28b5a37904d55bcca36003b9109aaa"; flake-utils.url = "github:numtide/flake-utils"; }; outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachDefaultSystem (system: let - pkgs = import nixpkgs { inherit system; }; + pkgs = import nixpkgs { + inherit system; + overlays = [ + (import ./nix/overlay/default.nix) + (import ./nix/overlay/development.nix) + (import ./nix/overlay/check.nix) + ]; + }; - # Build nixtamal package using the traditional nix-build approach - # This preserves the existing build infrastructure + # Build nixtamal package using callPackage nixtamalPkg = pkgs.callPackage ./nix/package/nixtamal.nix { - # Override dependencies as needed nixtamal = null; # Prevent infinite recursion }; diff --git a/lib/dune b/lib/dune index d0741bb..3d47618 100644 --- a/lib/dune +++ b/lib/dune @@ -10,8 +10,10 @@ jsont.bytesrw kdl logs + str saturn uri) + (flags (:standard -alert -deprecated)) (preprocess (pps ppx_deriving.enum diff --git a/lib/error.ml b/lib/error.ml index da9a271..ba92eac 100644 --- a/lib/error.ml +++ b/lib/error.ml @@ -25,6 +25,8 @@ type prefetch_method = [ | `Git | `Darcs | `Pijul + | `Nilla + | `Fossil ] [@@deriving show] @@ -34,6 +36,7 @@ type prefetch_error = [ | `JSON_parsing of prefetch_method * string | `Darcs_context of string | `Run_exception of prefetch_method * exn * string + | `Not_implemented of prefetch_method * string ] [@@deriving show] @@ -55,6 +58,7 @@ type error = [ | `Manifest of manifest_error | `Lockfile of lockfile_error | `Version_mismatch of string * string + | `Upgrade of string | `Input_foreman of input_foreman_error ] [@@deriving show] @@ -75,5 +79,7 @@ let pp ppf = function Fmt.(pf ppf "%a" pp_lockfile_error err) | `Version_mismatch (mnfst, lock) -> Fmt.pf ppf "Version mismatch: Manifest@@%s & Lockfile@@%s" mnfst lock - | `Input_foreman (`CouldNotAdd name) -> - Fmt.pf ppf "Could not set %a" Name.pp name + | `Upgrade msg -> + Fmt.pf ppf "Upgrade error: %s" msg + | `Input_foreman err -> + Fmt.(pf ppf "%a" pp_input_foreman_error err) diff --git a/lib/input.ml b/lib/input.ml index f2fb1a7..00ee2c9 100644 --- a/lib/input.ml +++ b/lib/input.ml @@ -137,6 +137,62 @@ module Pijul = struct [@@deriving show, eq, make, qcheck] end +(* Nilla is a Nix framework similar to flakes but with loaders and module system. + See: https://github.com/nilla-nix/nilla *) +module Nilla = struct + let default_path = Template.make "./nilla.nix" + + module Reference = struct + type t = [ + | `Branch of UTF8.t + | `Ref of UTF8.t + ] + [@@deriving show, eq, qcheck] + end + + type t = { + repository: Template.t; + mirrors: Template.t list; + reference: Reference.t; + datetime: UTF8.t option; (* ISO 8601 RFC 3339 *) + latest_revision: UTF8.t option; + path: Template.t; (* path to nilla.nix, default: ./nilla.nix *) + [@default default_path] + } + [@@deriving show, eq, make, qcheck] + + let default_latest_cmd nilla : Latest.Cmd.t = + let open Latest.Cmd in + let t = Template.make in + let git_ls_remote (ls_remote_args : Template.t list) : t = + let args = t "ls-remote" :: nilla.repository :: ls_remote_args in + ~${prog = t "git"; args} + |: {prog = t "cut"; args = [t "-f1"]} + in + match nilla.reference with + | `Branch b -> git_ls_remote [t "--heads"; t b] + | `Ref r -> git_ls_remote [t "--refs"; t r] +end + +module Fossil = struct + module Reference = struct + type t = [ + | `Branch of UTF8.t + | `Tag of UTF8.t + | `Checkin of UTF8.t + ] + [@@deriving show, eq, qcheck] + end + + type t = { + repository: Template.t; + reference: Reference.t; + date: UTF8.t option; + latest_checkin: UTF8.t option; + } + [@@deriving show, eq, make, qcheck] +end + module Hash = struct type algorithm = | SHA256 @@ -181,6 +237,8 @@ module Kind = struct | `Git of Git.t | `Darcs of Darcs.t | `Pijul of Pijul.t + | `Nilla of Nilla.t + | `Fossil of Fossil.t ] [@@deriving show, eq, qcheck] end @@ -200,6 +258,12 @@ let make_kind_pijul ~remote ?mirrors ~reference ?latest_state () = let make_kind_git ~repository ?mirrors ~reference ?latest_revision ?submodules ?lfs () = `Git (Git.make ~repository ?mirrors ~reference ?latest_revision ?submodules ?lfs ()) +let make_kind_nilla ~repository ?mirrors ~reference ?latest_revision ?path () = + `Nilla (Nilla.make ?mirrors ~repository ~reference ?latest_revision ?path ()) + +let make_kind_fossil ~repository ~reference ?date ?latest_checkin () = + `Fossil (Fossil.make ~repository ~reference ?date ?latest_checkin ()) + type t = { name: Name.t; kind: Kind.t; @@ -217,8 +281,9 @@ let latest_cmd (input : t) : Latest.Cmd.t option = ( match input.kind with | `Git g -> Some (Git.default_latest_cmd g) + | `Nilla n -> Some (Nilla.default_latest_cmd n) (* Would be nice if other tools did a better job letting you query the - remote repository directly, but that isn’t where we are *) + remote repository directly, but that isn't where we are *) | _ -> None ) | Some cmd -> Some cmd @@ -289,6 +354,36 @@ let jg_models2 (input : t) (needle : string) : Jingoo.Jg_types.tvalue = Option.iter (fun s -> Hashtbl.add htbl "state" (Tstr s)) p.latest_state; htbl end + | `Nilla n -> + begin + let htbl = make_hashtbl 4 in + ( + match n.reference with + | `Branch b -> Hashtbl.add htbl "branch" (Tstr b) + | `Ref r -> Hashtbl.add htbl "ref" (Tstr r) + ); + Option.iter (fun d -> Hashtbl.add htbl "datetime" (Tstr d)) n.datetime; + Hashtbl.add htbl "path" (Tstr (Template.take n.path)); + Option.iter + (fun r -> + List.iter (fun key -> Hashtbl.add htbl key (Tstr r)) ["rev"; "revision"] + ) + n.latest_revision; + htbl + end + | `Fossil f -> + begin + let htbl = make_hashtbl 3 in + ( + match f.reference with + | `Branch b -> Hashtbl.add htbl "branch" (Tstr b) + | `Tag t -> Hashtbl.add htbl "tag" (Tstr t) + | `Checkin c -> Hashtbl.add htbl "checkin" (Tstr c) + ); + Option.iter (fun d -> Hashtbl.add htbl "datetime" (Tstr d)) f.date; + Option.iter (fun c -> Hashtbl.add htbl "latest_checkin" (Tstr c)) f.latest_checkin; + htbl + end in match Hashtbl.find_opt hashtbl needle with | Some value -> value diff --git a/lib/input_foreman.ml b/lib/input_foreman.ml index dec3f32..9fefa5f 100644 --- a/lib/input_foreman.ml +++ b/lib/input_foreman.ml @@ -102,6 +102,33 @@ let pp_for_earthlings pff = Option.fold ~none: [] ~some: (fun d -> ["datetime", d]) p.datetime; Option.fold ~none: [] ~some: (fun s -> ["latest-state", s]) p.latest_state; ] + | `Nilla n -> + "nilla", + List.concat [ + ["repository", fill n.repository]; + (List.map (fun m -> "mirror", fill m) n.mirrors); + ( + match n.reference with + | `Branch b -> ["branch", b] + | `Ref r -> ["ref", r] + ); + ["path", fill n.path]; + Option.fold ~none: [] ~some: (fun d -> ["datetime", d]) n.datetime; + Option.fold ~none: [] ~some: (fun r -> ["latest-revision", r]) n.latest_revision; + ] + | `Fossil f -> + "fossil", + List.concat [ + ["repository", fill f.repository]; + ( + match f.reference with + | `Branch b -> ["branch", b] + | `Tag t -> ["tag", t] + | `Checkin c -> ["checkin", c] + ); + Option.fold ~none: [] ~some: (fun d -> ["date", d]) f.date; + Option.fold ~none: [] ~some: (fun c -> ["latest-checkin", c]) f.latest_checkin; + ] in let data_tuples : (string * string) list = List.concat [ @@ -346,6 +373,21 @@ let prefetch ~env ~proc_mgr ~name () : (unit, error) result = }, p_data.path ) + | `Nilla n, `Nilla n_data -> + Ok ( + {input with + kind = + `Nilla {n with + latest_revision = Some n_data.rev; + datetime = n_data.datetime; + }; + hash = {input.hash with + algorithm = n_data.hash.algorithm; + value = Some n_data.hash.value; + }; + }, + n_data.path + ) | _, _ -> failwith "Prefetch kind mismatch" end in @@ -448,6 +490,8 @@ let lock_one ~env ~sw ~proc_mgr ~force ~name : (unit, error) result = | `Git g -> Option.is_none g.latest_revision | `Darcs d -> Option.is_none d.latest_weak_hash | `Pijul p -> Option.is_none p.latest_state + | `Nilla n -> Option.is_none n.latest_revision + | `Fossil f -> Option.is_none f.latest_checkin in if needs_prefetch then prefetch ~env ~proc_mgr ~name () diff --git a/lib/lock_loader.ml b/lib/lock_loader.ml index c1648a0..3b6d128 100644 --- a/lib/lock_loader.ml +++ b/lib/lock_loader.ml @@ -16,6 +16,8 @@ module Features = struct let git = 1 lsl 2 let darcs = 1 lsl 3 let pijul = 1 lsl 4 + let nilla = 1 lsl 5 + let fossil = 1 lsl 6 let [@inline]has mask v = (mask land v) <> 0 let [@inline]add mask v = mask lor v @@ -30,6 +32,8 @@ module Features = struct | `Git _ -> add git | `Darcs _ -> add darcs | `Pijul _ -> add pijul + | `Nilla _ -> add nilla + | `Fossil _ -> add fossil let drop_input (input : Input.t) : t -> t = match input.kind with @@ -38,6 +42,8 @@ module Features = struct | `Git _ -> drop git | `Darcs _ -> drop darcs | `Pijul _ -> drop pijul + | `Nilla _ -> drop nilla + | `Fossil _ -> drop fossil end open Fmt diff --git a/lib/lockfile.ml b/lib/lockfile.ml index 36dd069..0792886 100644 --- a/lib/lockfile.ml +++ b/lib/lockfile.ml @@ -291,6 +291,77 @@ module Pijul = struct |> Object.finish end +module Nilla = struct + type t = { + repository: URI.t; + mirrors: URI.t list; + datetime: string option; + latest_revision: string option; + path: string; + } + [@@deriving show, eq, qcheck] + + let [@inline]to_lock + ~(models : Input.jg_models2) + ({repository; mirrors; datetime; latest_revision; path; _}: Input.Nilla.t) + : t + = + let to_uri = Fun.compose URI.of_string (Input.Template.fill ~models) in + { + repository = to_uri repository; + mirrors = List.map to_uri mirrors; + datetime; + latest_revision; + path = Input.Template.(fill ~models path); + } + + let jsont : t Jsont.t = + let open Jsont in + Object.map ~kind: "Nilla_lock" (fun repository mirrors datetime latest_revision path -> + {repository; mirrors; datetime; latest_revision; path} + ) + |> Object.mem "rp" URI.jsont ~enc: (fun i -> i.repository) + |> Object.mem "ms" (list URI.jsont) ~enc: (fun i -> i.mirrors) + |> Object.mem "dt" (option string) ~enc: (fun i -> i.datetime) + |> Object.mem "lr" (option string) ~enc: (fun i -> i.latest_revision) + |> Object.mem "pt" string ~enc: (fun i -> i.path) + |> Object.finish +end + +module Fossil = struct + type t = { + repository: URI.t; + mirrors: URI.t list; + datetime: string option; + latest_checkin: string option; + } + [@@deriving show, eq, qcheck] + + let [@inline]to_lock + ~(models : Input.jg_models2) + ({repository; reference; date; latest_checkin; _}: Input.Fossil.t) + : t + = + let to_uri = Fun.compose URI.of_string (Input.Template.fill ~models) in + { + repository = to_uri repository; + mirrors = []; (* Fossils don't have mirrors in upstream, so empty *) + datetime = date; + latest_checkin; + } + + let jsont : t Jsont.t = + let open Jsont in + Object.map ~kind: "Fossil_lock" (fun repository mirrors datetime latest_checkin -> + {repository; mirrors; datetime; latest_checkin} + ) + |> Object.mem "rp" URI.jsont ~enc: (fun i -> i.repository) + |> Object.mem "ms" (list URI.jsont) ~enc: (fun i -> i.mirrors) + |> Object.mem "dt" (option string) ~enc: (fun i -> i.datetime) + |> Object.mem "lc" (option string) ~enc: (fun i -> i.latest_checkin) + |> Object.finish +end + module Kind = struct type t = [ | `File of File.t @@ -298,6 +369,8 @@ module Kind = struct | `Git of Git.t | `Darcs of Darcs.t | `Pijul of Pijul.t + | `Nilla of Nilla.t + | `Fossil of Fossil.t ] [@@deriving show, eq, qcheck] @@ -307,6 +380,8 @@ module Kind = struct | `Git g -> `Git (Git.to_lock ~models g) | `Darcs d -> `Darcs (Darcs.to_lock ~models d) | `Pijul p -> `Pijul (Pijul.to_lock ~models p) + | `Nilla n -> `Nilla (Nilla.to_lock ~models n) + | `Fossil f -> `Fossil (Fossil.to_lock ~models f) let jsont : t Jsont.t = let open Jsont in @@ -316,6 +391,8 @@ module Kind = struct | `Git g -> encode_tag 2 Git.jsont g | `Darcs d -> encode_tag 3 Darcs.jsont d | `Pijul p -> encode_tag 4 Pijul.jsont p + | `Nilla n -> encode_tag 5 Nilla.jsont n + | `Fossil f -> encode_tag 6 Fossil.jsont f and dec = function | [|tag; value|] -> begin @@ -335,6 +412,12 @@ module Kind = struct | 4 -> Json.decode' Pijul.jsont value |> Result.map (fun v -> `Pijul v) + | 5 -> + Json.decode' Nilla.jsont value + |> Result.map (fun v -> `Nilla v) + | 6 -> + Json.decode' Fossil.jsont value + |> Result.map (fun v -> `Fossil v) | n -> Error.msgf Meta.none "Unknown reference enum tag: %d" n ) with diff --git a/lib/manifest.ml b/lib/manifest.ml index 1f0f255..77c75f3 100644 --- a/lib/manifest.ml +++ b/lib/manifest.ml @@ -379,6 +379,147 @@ module Pijul = struct } end +module Nilla = struct + module Reference = struct + type t = Input.Nilla.Reference.t + [@@deriving show, eq, qcheck] + + let codec : t KDL.codec = { + to_kdl = (fun ref -> + let open KDL in + match ref with + | `Branch b -> [KDL.node "branch" ~args: [arg (`String b)] []] + | `Ref r -> [KDL.node "ref" ~args: [arg (`String r)] []] + ); + of_kdl = (fun kdl -> + let open KDL.L in + let open KDL.Valid in + let node_names = ["branch"; "ref"] + and branch = ll @@ kdl.@(node "branch" // arg 0 // string_value) + and ref = ll @@ kdl.@(node "ref" // arg 0 // string_value) + in + match branch, ref with + | Ok b, Error _ -> Ok (`Branch b) + | Error _, Ok r -> Ok (`Ref r) + | Error _, Error _ -> Error [`OneRequired node_names] + | _, _ -> Error [`OnlyOneOf node_names] + ); + } + end + + type t = { + repository: Template.t; + mirrors: Template.t list; + reference: Reference.t; + path: Template.t; + } + [@@deriving show, eq, make, qcheck] + + let [@inline]to_manifest ({repository; mirrors; reference; path; _}: Input.Nilla.t) : t = + make ~repository ~mirrors ~reference ~path () + + let [@inline]of_manifest ({repository; mirrors; reference; path}: t) : Input.Nilla.t = + Input.Nilla.make ~repository ~mirrors ~reference ~path () + + let codec : t KDL.codec = { + to_kdl = (fun nilla -> + let open KDL in + let repository = + node "repository" ~args: [Template.to_arg nilla.repository] [] + and path = + node "path" ~args: [Template.to_arg nilla.path] [] + and nodes = + Reference.codec.to_kdl nilla.reference + in + let nodes = + if List.is_empty nilla.mirrors then + nodes + else + node "mirrors" ~args: (List.map Template.to_arg nilla.mirrors) [] :: nodes + in + let nodes = path :: repository :: nodes in + [node "nilla" nodes] + ); + of_kdl = (fun kdl -> + let open KDL.L in + let open KDL.Valid in + let* nilla = ll @@ kdl.@(node "nilla") in + let+ repository = Template.of_child ~name: "repository" nilla + and+ mirrors = Template.of_mirrors nilla + and+ reference = Reference.codec.of_kdl nilla.children + and+ path = Template.of_child ~name: "path" nilla + in + {repository; mirrors; reference; path} + ); + } +end + +module Fossil = struct + module Reference = struct + type t = Input.Fossil.Reference.t + [@@deriving show, eq, qcheck] + + let codec : t KDL.codec = { + to_kdl = (fun ref -> + let open KDL in + match ref with + | `Branch c -> [KDL.node "branch" ~args: [arg (`String c)] []] + | `Tag s -> [KDL.node "tag" ~args: [arg (`String s)] []] + | `Checkin c -> [KDL.node "check-in" ~args: [arg (`String c)] []] + ); + of_kdl = (fun kdl -> + let open KDL.L in + let open KDL.Valid in + let node_names = ["branch"; "tag"; "check-in"] + and branch = ll @@ kdl.@(node "branch" // arg 0 // string_value) + and tag = ll @@ kdl.@(node "tag" // arg 0 // string_value) + and checkin = ll @@ kdl.@(node "check-in" // arg 0 // string_value) + in + match branch, tag, checkin with + | Ok b, Error _, Error _ -> Ok (`Branch b) + | Error _, Ok t, Error _ -> Ok (`Tag t) + | Error _, Error _, Ok c -> Ok (`Checkin c) + | Error _, Error _, Error _ -> Error [`OneRequired node_names] + | _, _, _ -> Error [`OnlyOneOf node_names] + ); + } + end + + type t = { + repository: Template.t; + reference: Reference.t; + } + [@@deriving show, eq, make, qcheck] + + let [@inline]to_manifest ({repository; reference; _}: Input.Fossil.t) : t = + make ~repository ~reference + + let [@inline]of_manifest ({repository; reference}: t) : Input.Fossil.t = + Input.Fossil.make ~repository ~reference () + + let codec : t KDL.codec = { + to_kdl = (fun fossil -> + let open KDL in + let repository = + node "repository" ~args: [Template.to_arg fossil.repository] [] + and nodes = + Reference.codec.to_kdl fossil.reference + in + let nodes = repository :: nodes in + [node "fossil" nodes] + ); + of_kdl = (fun kdl -> + let open KDL.L in + let open KDL.Valid in + let* fossil = ll @@ kdl.@(node "fossil") in + let+ repository = Template.of_child ~name: "repository" fossil + and+ reference = Reference.codec.of_kdl fossil.children + in + {repository; reference} + ); + } +end + module Kind = struct type t = [ | `File of File.t @@ -386,6 +527,8 @@ module Kind = struct | `Git of Git.t | `Darcs of Darcs.t | `Pijul of Pijul.t + | `Nilla of Nilla.t + | `Fossil of Fossil.t ] [@@deriving show, eq, qcheck] @@ -395,6 +538,8 @@ module Kind = struct | `Git g -> `Git (Git.to_manifest g) | `Darcs d -> `Darcs (Darcs.to_manifest d) | `Pijul p -> `Pijul (Pijul.to_manifest p) + | `Nilla n -> `Nilla (Nilla.to_manifest n) + | `Fossil f -> `Fossil (Fossil.to_manifest f) let of_manifest : t -> Input.Kind.t = function | `File f -> `File (File.of_manifest f) @@ -402,6 +547,8 @@ module Kind = struct | `Git g -> `Git (Git.of_manifest g) | `Darcs d -> `Darcs (Darcs.of_manifest d) | `Pijul p -> `Pijul (Pijul.of_manifest p) + | `Nilla n -> `Nilla (Nilla.of_manifest n) + | `Fossil f -> `Fossil (Fossil.of_manifest f) let codec : t KDL.codec = { to_kdl = (function @@ -410,27 +557,35 @@ module Kind = struct | `Git g -> Git.codec.to_kdl g | `Darcs d -> Darcs.codec.to_kdl d | `Pijul p -> Pijul.codec.to_kdl p + | `Nilla n -> Nilla.codec.to_kdl n + | `Fossil f -> Fossil.codec.to_kdl f ); of_kdl = (fun kdl -> - let kind_names = ["file"; "archive"; "git"; "darcs"; "pijul"] in + let kind_names = ["file"; "archive"; "git"; "darcs"; "pijul"; "nilla"; "fossil"] in match File.codec.of_kdl kdl, Archive.codec.of_kdl kdl, Git.codec.of_kdl kdl, Darcs.codec.of_kdl kdl, - Pijul.codec.of_kdl kdl with - | Ok file, Error _, Error _, Error _, Error _ -> + Pijul.codec.of_kdl kdl, + Nilla.codec.of_kdl kdl, + Fossil.codec.of_kdl kdl with + | Ok file, Error _, Error _, Error _, Error _, Error _, Error _ -> Ok (`File file) - | Error _, Ok archive, Error _, Error _, Error _ -> + | Error _, Ok archive, Error _, Error _, Error _, Error _, Error _ -> Ok (`Archive archive) - | Error _, Error _, Ok git, Error _, Error _ -> + | Error _, Error _, Ok git, Error _, Error _, Error _, Error _ -> Ok (`Git git) - | Error _, Error _, Error _, Ok darcs, Error _ -> + | Error _, Error _, Error _, Ok darcs, Error _, Error _, Error _ -> Ok (`Darcs darcs) - | Error _, Error _, Error _, Error _, Ok pijul -> + | Error _, Error _, Error _, Error _, Ok pijul, Error _, Error _ -> Ok (`Pijul pijul) - | Error _, Error _, Error _, Error _, Error _ -> + | Error _, Error _, Error _, Error _, Error _, Ok nilla, Error _ -> + Ok (`Nilla nilla) + | Error _, Error _, Error _, Error _, Error _, Error _, Ok fossil -> + Ok (`Fossil fossil) + | Error _, Error _, Error _, Error _, Error _, Error _, Error _ -> Error [`OneRequired kind_names] - | _, _, _, _, _ -> + | _, _, _, _, _, _, _ -> Error [`OnlyOneOf kind_names] ); } diff --git a/lib/nixtamal.ml b/lib/nixtamal.ml index 9ceaa36..a9435ab 100644 --- a/lib/nixtamal.ml +++ b/lib/nixtamal.ml @@ -10,6 +10,7 @@ module Input = Input module Input_foreman = Input_foreman module Working_directory = Working_directory module KDL = KDL +module Schema = Schema type error = Error.error @@ -34,6 +35,10 @@ let meld_input_with_lock (input : Input.t) (lock : Lockfile.Input'.t) : Input.t } | `Pijul pijul, `Pijul({datetime; latest_state; _}: Lockfile.Pijul.t) -> `Pijul {pijul with datetime; latest_state} + | `Nilla nilla, `Nilla({datetime; latest_revision; _}: Lockfile.Nilla.t) -> + `Nilla {nilla with datetime; latest_revision} + | `Fossil fossil, `Fossil({datetime; latest_checkin; _}: Lockfile.Fossil.t) -> + `Fossil {fossil with date = datetime; latest_checkin} | _, _ -> failwith "Input kind mismatch." ); hash = {input.hash with value = lock.hash.value}; @@ -63,6 +68,16 @@ let read_manifest_and_lockfile () : (Name.Name.t list, error) result = | Some lock when not (String.equal manifest.version lock.version) -> Error (`Version_mismatch (manifest.version, lock.version)) | _ -> + let lockfile_opt = match lockfile_opt with + | Some lock -> Some lock + | None -> + Logs.info (fun m -> m "Lockfile missing, creating new empty lockfile"); + match Lockfile.make ~version: manifest.version () with + | Ok lock -> Some lock + | Error e -> + Logs.warn (fun m -> m "Failed to create lockfile: %a" Error.pp_lockfile_error e); + None + in let to_input d = let input = Manifest.Input'.of_manifest d in let found_input = @@ -203,3 +218,149 @@ let refresh ~env ~domain_count ?names () : (unit, error) result = Lock_loader.write (); Input_foreman.clean_unlisted_from_silo (); Ok () + +let backup_path (filename : string) : string = filename ^ ".bak" + +let upgrade ?from ?(to_ = Schema.Version.current) ?(dry_run = false) () : (unit, error) result = + let (let*) = Result.bind in + let working_dir = Working_directory.get () in + let lockfile_path = Eio.Path.(working_dir / Lockfile.filename) in + let manifest_path = Eio.Path.(working_dir / Manifest.filename) in + let manifest_content = + Eio.Path.with_open_in manifest_path @@ fun flow -> + let buf = Eio.Buf_read.of_flow flow ~max_size: max_int in + Eio.Buf_read.take_all buf + in + Logs.info (fun m -> m "Current schema version: %a" Schema.Version.pp Schema.Version.current); + let* manifest_version = + let open KDL.L in + let open KDL.Valid in + match Eio.Path.with_open_in manifest_path KDL.of_flow with + | Error _ -> Error (`Manifest `Not_set_up) + | Ok kdl_doc -> + match ll @@ kdl_doc.@(node "version" // arg 0 // string_value) with + | Error _ -> Error (`Upgrade "Manifest version not found") + | Ok version_str -> + match Schema.Version.of_string version_str with + | Some v -> Ok v + | None -> Error (`Upgrade (Fmt.str "Unknown manifest schema version: %s" version_str)) + in + Logs.info (fun m -> m "Manifest version: %a" Schema.Version.pp manifest_version); + let* lockfile_version = + match Lockfile.read () with + | Ok (Some lock) -> + begin match Schema.Version.of_string lock.version with + | Some v -> Ok v + | None -> Error (`Upgrade (Fmt.str "Unknown lockfile schema version: %s" lock.version)) + end + | Ok None -> Error (`Upgrade "Lockfile missing, cannot determine version") + | Error err -> Error (`Lockfile (`Parsing err)) + in + Logs.info (fun m -> m "Lockfile version: %a" Schema.Version.pp lockfile_version); + let* () = + match from with + | None -> Ok () + | Some from' when from' <> manifest_version -> + Error (`Version_mismatch (Schema.Version.to_string manifest_version, Schema.Version.to_string from')) + | Some from' when from' <> lockfile_version -> + Error (`Version_mismatch (Schema.Version.to_string lockfile_version, Schema.Version.to_string from')) + | Some _ -> Ok () + in + let needs_manifest_upgrade = manifest_version < to_ + and needs_lock_upgrade = lockfile_version < to_ + in + if not needs_lock_upgrade && not needs_manifest_upgrade then begin + Logs.app (fun m -> m "Already at %a" Schema.Version.pp to_); + Ok () + end else if dry_run then begin + if needs_manifest_upgrade then + Logs.app (fun m -> + m "Would upgrade %s: %a → %a" + Manifest.filename Schema.Version.pp manifest_version Schema.Version.pp to_ + ); + if needs_lock_upgrade then + Logs.app (fun m -> + m "Would upgrade %s: %a → %a" + Lockfile.filename Schema.Version.pp lockfile_version Schema.Version.pp to_ + ); + Logs.app (fun m -> m "No changes applied."); + Ok () + end + else + let manifest_backup_path = Eio.Path.(working_dir / backup_path Manifest.filename) + and lock_backup_path = Eio.Path.(working_dir / backup_path Lockfile.filename) + and manifest_backup_created = ref false + and lock_backup_created = ref false + in + let cleanup_backups () = + Logs.info (fun m -> m "Cleaning up backups…"); + if !manifest_backup_created then + try Eio.Path.unlink manifest_backup_path + with _ -> Logs.err (fun m -> m "Failed to cleanup manifest backup"); + if !lock_backup_created then + try Eio.Path.unlink lock_backup_path + with _ -> Logs.err (fun m -> m "Failed to cleanup lock backup") + in + let rollback () = + Logs.info (fun m -> m "Rolling back backups…"); + if !manifest_backup_created then + try Eio.Path.rename manifest_backup_path manifest_path + with _ -> Logs.err (fun m -> m "Failed to rollback manifest"); + if !lock_backup_created then + try Eio.Path.rename lock_backup_path lockfile_path + with _ -> Logs.err (fun m -> m "Failed to rollback lock"); + cleanup_backups () + in + try + let* () = + if needs_lock_upgrade then + match Lockfile.read () with + | Error err -> Error (`Lockfile (`Parsing err)) + | Ok lockfile_opt -> + match lockfile_opt with + | None -> Error (`Upgrade "Lockfile not set up for upgrade") + | Some lockfile -> + Logs.app (fun m -> m "Upgrading %s: %a → %a" Lockfile.filename Schema.Version.pp lockfile_version Schema.Version.pp to_); + let upgraded = {lockfile with version = Schema.Version.to_string to_} in + Lockfile.lockfile := Some upgraded; + Eio.Path.rename lockfile_path lock_backup_path; + match Lockfile.write () with + | Error err -> Error (`Lockfile err) + | Ok () -> + lock_backup_created := true; + Ok () + else Ok () + in + let* () = + if needs_manifest_upgrade then + Error.tag_manifest @@ begin + Logs.app (fun m -> + m "Upgrading %s: %a → %a" + Manifest.filename Schema.Version.pp manifest_version Schema.Version.pp to_ + ); + let upgraded_content = Str.global_replace (Str.regexp "0\\.1\\.1") (Schema.Version.to_string to_) manifest_content in + Eio.Path.rename manifest_path manifest_backup_path; + Eio.Path.with_open_out ~create:(`Or_truncate 0o644) manifest_path @@ fun flow -> + Eio.Flow.copy_string upgraded_content flow; + manifest_backup_created := true; + Ok () + end + else Ok () + in + let* () = + match Manifest.read () with + | Ok _ -> Logs.info (fun m -> m "Manifest verified."); Ok () + | Error e -> Error (`Manifest (`Parsing [`ParseError e])) + in + let* () = + match Lockfile.read () with + | Ok _ -> Logs.info (fun m -> m "Lockfile verified."); Ok () + | Error e -> Error (`Lockfile (`Parsing e)) + in + cleanup_backups (); + Ok () + with + | exn -> + Logs.err (fun m -> m "Upgrade failed: %s" (Printexc.to_string exn)); + rollback (); + Error (`Upgrade "Failed") diff --git a/lib/prefetch.ml b/lib/prefetch.ml index f4923f5..eb65e60 100644 --- a/lib/prefetch.ml +++ b/lib/prefetch.ml @@ -50,6 +50,17 @@ module Hash = struct |> Object.opt_mem "blake3" string |> Object.opt_mem "sha256" string |> Object.opt_mem "sha512" string + + (* Parse SRI hash like "sha256-XpTlkQ6FXaK0SgN0bu/yji2NASPDuseVaO5bgdFROkM=" *) + let make_from_SRI_hash (value : string) : t = + match String.split_on_char '-' value with + | [algo_str; sri_value] -> + ( + match Input.Hash.algorithm_of_string algo_str with + | Some algorithm -> {algorithm; value = sri_value} + | None -> Jsont.Error.msgf Jsont.Meta.none "Unsupported hash algorithm: %s" algo_str + ) + | _ -> Jsont.Error.msgf Jsont.Meta.none "Invalid SRI hash format: %s" value end module File = struct @@ -264,6 +275,106 @@ module Pijul = struct |> Result.map_error (fun err -> `JSON_parsing (method', err)) end +module Nilla = struct + type t = { + datetime: string option; + path: string; + rev: string; + hash: Hash.t; + } + [@@deriving make, show] + + let jsont : t Jsont.t = + let open Jsont in + Object.map + ~kind: "Prefetch_Nilla" + (fun path datetime rev blake3 sha256 sha512 -> + let hash = Hash.make_from_opts blake3 sha256 sha512 in + make ~path ?datetime ~rev ~hash () + ) + |> Object.mem "path" string ~enc: (fun i -> i.path) + |> Object.opt_mem "date" string ~enc: (fun i -> i.datetime) + |> Object.mem "rev" string ~enc: (fun i -> i.rev) + |> Hash.add_jsont_case + |> Object.finish + + let latest_cmd (n : Input.Nilla.t) ~models = + let cmd = [ + "nix-prefetch-git"; + "--no-deepClone"; + "--quiet"; + "--url"; + URI.to_string (URI.of_string (Input.Template.fill n.repository ~models)); + ] + in + List.concat [ + cmd; + ( + match n.reference with + | `Branch b -> ["--branch-name"; b] + | `Ref r -> ["--rev"; r] + ); + ] + + let get_latest ~proc_mgr ~proc_env (n : Input.Nilla.t) ~models = + let (let*) = Result.bind in + let method' = `Nilla in + let* stdout = run_and_gather ~proc_mgr ~proc_env method' (latest_cmd n ~models) in + let* parsed = Jsont_bytesrw.decode_string jsont stdout + |> Result.map_error (fun err -> `JSON_parsing (method', err)) + in + let nilla_path = Input.Template.fill n.path ~models in + let full_path = Filename.concat parsed.path nilla_path in + if Sys.file_exists full_path then + Ok {parsed with path = full_path} + else + Error (`Bad_output (method', Printf.sprintf "nilla.nix not found at path '%s' in repository" nilla_path)) +end + +module Fossil = struct + type t = { + path: string; + datetime: string option; + checkin: string; + hash: Hash.t + } + [@@deriving make, show] + + let jsont : t Jsont.t = + let open Jsont in + Object.map + ~kind: "Prefetch_Fossil" + (fun path datetime checkin hash_str -> + let hash = Hash.make_from_SRI_hash hash_str in + make ~path ?datetime ~checkin ~hash () + ) + |> Object.mem "path" string ~enc: (fun i -> i.path) + |> Object.opt_mem "date" string ~enc: (fun i -> i.datetime) + |> Object.mem "rev" string ~enc: (fun i -> i.checkin) + |> Object.mem "hash" string + |> Object.finish + + let latest_cmd (f : Input.Fossil.t) ~models = + let cmd = [ + "nix-prefetch-fossil"; + "--url"; + URI.to_string (URI.of_string (Input.Template.fill f.repository ~models)); + ] + in + cmd @ + match f.reference with + | `Branch b -> ["--rev"; b] + | `Tag t -> ["--rev"; t] + | `Checkin c -> ["--rev"; c] + + let get_latest ~proc_mgr ~proc_env (f : Input.Fossil.t) ~models = + let (let*) = Result.bind in + let method' = `Fossil in + let* stdout = run_and_gather ~proc_mgr ~proc_env method' (latest_cmd f ~models) in + Jsont_bytesrw.decode_string jsont stdout + |> Result.map_error (fun err -> `JSON_parsing (method', err)) +end + type prefetch_kind_result = ( [ | `File of File.t @@ -271,6 +382,8 @@ type prefetch_kind_result = ( | `Git of Git.t | `Darcs of Darcs.t | `Pijul of Pijul.t + | `Nilla of Nilla.t + | `Fossil of Fossil.t ], error ) result @@ -298,3 +411,9 @@ let get_latest ~env ~proc_mgr (input : Input.t) : prefetch_kind_result = | `Pijul pijul -> Pijul.get_latest ~proc_mgr ~proc_env pijul ~models |> Result.map (fun p -> `Pijul p) + | `Nilla nilla -> + Nilla.get_latest ~proc_mgr ~proc_env nilla ~models + |> Result.map (fun n -> `Nilla n) + | `Fossil fossil -> + Fossil.get_latest ~proc_mgr ~proc_env fossil ~models + |> Result.map (fun f -> `Fossil f) diff --git a/lib/schema.ml b/lib/schema.ml new file mode 100644 index 0000000..40b058a --- /dev/null +++ b/lib/schema.ml @@ -0,0 +1,31 @@ +(*─────────────────────────────────────────────────────────────────────────────┐ +│ SPDX-FileCopyrightText: 2025 toastal │ +│ SPDX-License-Identifier: LGPL-2.1-or-later WITH OCaml-LGPL-linking-exception │ +└─────────────────────────────────────────────────────────────────────────────*) +module Version = struct + type t = + | V0_1_1 + | V0_2_0 + [@@deriving show, enum, eq, ord] + + let of_string s = + match s with + | "0.1.1" -> Some V0_1_1 + | "0.2.0" -> Some V0_2_0 + | _ -> None + + let to_string = function + | V0_1_1 -> "0.1.1" + | V0_2_0 -> "0.2.0" + + let current : t = Option.get (of_enum max) + + let versions : t array = + let vs = Dynarray.create () in + for idx = min to max do + match of_enum idx with + | Some v -> Dynarray.add_last vs v + | None -> () + done; + Dynarray.to_array vs +end \ No newline at end of file diff --git a/llm/2026-01-07-Session-Changes-Summary.md b/llm/2026-01-07-Session-Changes-Summary.md deleted file mode 100644 index d7ed6e3..0000000 --- a/llm/2026-01-07-Session-Changes-Summary.md +++ /dev/null @@ -1,33 +0,0 @@ -# 2026-01-07-Session-Changes-Summary - -## Summary -This note summarizes all changes made during the development session for the Nixtamal project, from initial darcs cloning to final git commit with comprehensive documentation. - -## What Was Changed -1. **Project Conversion**: Migrated Nixtamal from Darcs to Git, preserving full history (140 commits, 9 tags) -2. **Documentation Creation**: Created AGENTS.md with coding guidelines for AI agents -3. **LLM Infrastructure**: Established ./llm/ folder for agent notes with README and template -4. **README Updates**: Updated README.rst with current info and converted to README.asciidoc -5. **Build Verification**: Confirmed builds and tests work correctly -6. **Comprehensive Commit**: Created a detailed commit message documenting the entire project and session changes - -## Why This Change -The session aimed to modernize the project's version control, enhance documentation for AI collaboration, and establish infrastructure for future development tracking. This ensures the project is accessible to broader development tools while maintaining its core functionality. - -## Challenges & Solutions -- **Darcs to Git Conversion**: Challenge was ensuring complete history preservation. Solution: Used darcs convert export piped to git fast-import, verified commit/tag counts. -- **Documentation Accuracy**: README had outdated info. Solution: Updated with current data and converted to AsciiDoc for better compatibility. -- **Agent Guidelines**: No existing standards for AI coding. Solution: Created comprehensive AGENTS.md based on codebase analysis. -- **Note Infrastructure**: Needed system for tracking learnings. Solution: Created ./llm/ with standardized template. - -## Learnings -- Darcs conversion to Git is reliable using standard tools -- AsciiDoc provides better markup flexibility than RST for technical docs -- Establishing agent guidelines early improves consistency -- Comprehensive commit messages serve as excellent project documentation -- Nix-based builds require careful environment management - -## Related -- Files: AGENTS.md, README.asciidoc, llm/README.md, llm/2026-01-07-Session-Changes-Summary.md -- Commits: 7b55727 (comprehensive commit) -- Issues: None (session work) \ No newline at end of file diff --git a/llm/2026-02-07-attribution-audit.md b/llm/2026-02-07-attribution-audit.md deleted file mode 100644 index e64d596..0000000 --- a/llm/2026-02-07-attribution-audit.md +++ /dev/null @@ -1,25 +0,0 @@ -# Documentation Attribution Audit Completion - -## Summary -Completed comprehensive audit of nixtamal documentation to ensure proper attribution to toastal as the original author and project lead, maintaining voice consistency and attribution standards. - -## What Was Changed -- Modified AGENTS.md strategic position statement (line 6) to attribute dual workflow implementation to toastal instead of using impersonal "we're implementing". - -## Why This Change -- Ensured voice attribution requirements are met by speaking from toastal's perspective -- Maintained consistency with project's attribution standards -- Enhanced transparency and proper credit to original author - -## Challenges & Solutions -No significant challenges encountered - all other documentation already properly credited toastal, requiring only a single line modification in AGENTS.md. - -## Learnings -- Importance of consistent voice attribution in documentation -- Value of maintaining clear author identification across all project materials -- Benefits of regular attribution audits for open source projects - -## Related -- Files: AGENTS.md -- Timestamp: 2026-02-07 -- Verification: All documentation now compliant with attribution requirements \ No newline at end of file diff --git a/llm/2026-02-07-phase1-flake-integration.md b/llm/2026-02-07-phase1-flake-integration.md deleted file mode 100644 index 2ceafe2..0000000 --- a/llm/2026-02-07-phase1-flake-integration.md +++ /dev/null @@ -1,133 +0,0 @@ -# Phase 1 Flake Integration - Strategic Implementation - -## Summary -Successfully implemented Phase 1 of flake integration for nixtamal using a dual-support architecture that preserves the project's core mission while enabling flake ecosystem compatibility. The solution maintains philosophical consistency as a flake complement rather than replacement. - -## What Was Changed - -### Core Implementation -- **Created `flake.nix`**: Implemented wrapper pattern that directly uses the existing `nix/package/nixtamal.nix` build definition -- **Generated `flake.lock`**: Automatically created lock file for reproducible builds -- **Updated `AGENTS.md`**: Added comprehensive Phase 1 planning documentation and strategic architecture details - -### Flake Outputs Implemented -- **packages.{system}**: `nixtamal` and `default` packages -- **devShells.{system}.default**: Development environment using mkShell -- **checks.{system}**: Basic build verification check -- **lib**: Ecosystem bridge functions for hybrid workflows -- **apps**: Direct application interface for running nixtamal -- **legacyPackages**: Backward compatibility layer - -## Why This Change - -### Strategic Positioning -- **Maintains Core Mission**: Nixtamal remains philosophically an alternative to flakes -- **Ecosystem Bridge**: Provides flake interface without compromising existing workflows -- **Incremental Adoption**: Enables gradual migration paths for flake users -- **Market Expansion**: Access to growing flake ecosystem while maintaining differentiation - -### Technical Benefits -- **Preserves Existing Infrastructure**: No breaking changes to current Nix build system -- **Enables Hybrid Workflows**: Flakes for outer layer, nixtamal for inner dependency management -- **Developer Experience**: IDE integration and modern tooling support -- **CI/CD Integration**: Compatibility with flake-based build pipelines - -## Challenges & Solutions - -### Challenge: Pure Evaluation Context -**Problem**: `builtins.currentSystem` not available in flake evaluation -**Solution**: Used direct `pkgs.callPackage` approach that bypasses the need for system context - -### Challenge: Complex Release Infrastructure -**Problem**: `release.nix` has complex dependency on nixtamal's own input system -**Solution**: Simplified approach using direct package definition rather than wrapping complex release logic - -### Challenge: Maintaining Philosophical Consistency -**Problem**: Risk of undermining nixtamal's positioning as flake alternative -**Solution**: Implemented wrapper pattern that positions flakes as interface layer, not replacement - -## Technical Architecture - -### Design Pattern: Wrapper Interface -```nix -# Flake provides interface layer over existing infrastructure -nixtamalPkg = pkgs.callPackage ./nix/package/nixtamal.nix { - nixtamal = null; # Prevent infinite recursion -}; -``` - -### Hy