================================================================================ Nixtamal Development History ================================================================================ .. role:: ab .. role:: ac ------------------------------------------------------------------------------ Origins (2024) ------------------------------------------------------------------------------ **The Problem** In 2024, toastal was working with Nix in environments that required diverse version control systems. The Nix ecosystem at the time was heavily Git-centric: - Flakes only supported Git repositories well - Darcs and Pijul projects required manual workarounds - No standardized way to pin non-Git dependencies - Existing tools (niv, npins) had limited VCS support **The First Spark** The initial idea was simple: a version pinning tool that treats all VCS equally. Not Git-first with other VCS as afterthoughts, but truly first-class support for the diversity of version control systems. **Technology Choice** OCaml was chosen for several reasons: - Strong type system for complex domain modeling - Excellent concurrency support (Eio library) - Decades of stability and maintainability - Personal familiarity - Nix integration already well-established KDL was chosen for configuration because: - More readable than JSON - More structured than TOML - No YAML whitespace hell - Native comment support (important for hand-edited files) ------------------------------------------------------------------------------ Early Development (2024-2025) ------------------------------------------------------------------------------ **Darcs-First Approach** The project was initially developed in Darcs, hosted at: ``https://darcs.toastal.in.th/nixtamal/trunk`` This was a deliberate choice: - Dogfooding the VCS diversity principle - Darcs' patch theory is elegant - Self-hosting independence **Version 0.1.x Series** The early versions focused on establishing the core: - 0.1.0 - Basic manifest/lockfile structure - 0.1.1 - Stabilized schema with proper versioning - Git, Darcs, and Pijul support added iteratively - File and Archive inputs for completeness **Key Technical Decisions:** 1. **Concurrent prefetching** using OCaml 5's Eio library 2. **Template system** using Jingoo for dynamic URLs 3. **KDL codecs** for type-safe configuration parsing 4. **Lockfile format** as JSON (machine-readable, diffable) ------------------------------------------------------------------------------ The Nilla Integration (2025) ------------------------------------------------------------------------------ **What is Nilla?** Nilla (https://github.com/nilla-nix/nilla) is a Nix framework with loaders and a module system. It represents a new approach to Nix project structure. **Integration Work** Adding Nilla support required: - New input type for Nilla references - Understanding Nilla's loader architecture - Manifest serialization for Nilla inputs - Lockfile integration **Status:** Types, codecs, and manifest serialization implemented. Prefetch support deferred pending Nilla ecosystem maturation. **Impact:** Nixtamal can now manage Nilla-based inputs alongside traditional Nix dependencies, supporting hybrid workflows. ------------------------------------------------------------------------------ The Git Migration (2025-2026) ------------------------------------------------------------------------------ **Why Migrate?** Despite philosophical alignment with Darcs, practical realities emerged: 1. **Contributor friction** - Most developers know Git, not Darcs 2. **Tooling gaps** - CI/CD, GitHub Actions, etc. built around Git 3. **Visibility** - Git repositories get more community attention 4. **Accessibility** - Easier for contributors to submit patches **The Fork** A community-maintained Git fork was created. The original Darcs repository continues to exist as the "source of truth" with the Git repository as a mirror with additional development. **Maintaining Principles:** The migration didn't compromise core principles: - Still self-hosted (not GitHub) - XMPP remains primary communication - Website still independent - Philosophy document unchanged ------------------------------------------------------------------------------ The Flake Integration (2025-2026) ------------------------------------------------------------------------------ **The Controversy** Nix flakes are controversial in the community: - Pro: Integrated locking, caching, composability - Con: JSON configuration, Git lockfiles, centralized registry **The Compromise** Rather than choosing sides, Nixtamal embraced both: **Traditional Nix** (Default) - KDL configuration (human-readable) - External lockfiles - Works with ``nix-build``, ``nix-shell`` **Flake Integration** (Optional) - ``nixtamal.lib`` for flake consumption - Hybrid workflows supported - Use nixtamal projects as flake inputs **Technical Implementation:** - ``flake.nix`` with proper overlay system - ``nix/tamal/`` directory for lockfile consumption - Library functions for ecosystem integration - Phase 1 implementation notes documented **Outcome:** Users choose their workflow. No forced migration. ------------------------------------------------------------------------------ Major Upstream Port (2026-04-14) ------------------------------------------------------------------------------ **The Gap** The Git fork had diverged from upstream Darcs: - Git fork: 145 commits, ~0.3.x-alpha - Darcs upstream: 347 patches, v1.1.2 - ~202 patches behind - Missing: schema upgrade, Fossil VCS, TUI, etc. **The Port** A systematic porting effort was undertaken: **Phase 1 - Critical Fixes:** - Cmdliner 2.x compatibility (variable shadowing fixes) - Lockfile auto-creation (don't error on missing lockfile) **Phase 2 - Major Features:** - Schema upgrade command with backup/rollback - Fossil VCS support (complete implementation) - Hash SRI parsing improvements - Error type refinements **Challenges:** 1. **Type compatibility** - Error types evolved between versions 2. **Module renames** - Foreman vs input_foreman 3. **Pattern match completeness** - Fossil added to all pattern matches 4. **Flake preservation** - Must maintain existing flake integration **Resolution:** Bellana-Sugar and Bellana-Drone collaborated to resolve type issues while preserving the Git fork's flake additions. All tests passing. **Commit:** ``4ed722f8be5a3bb5ea8ffdfe89f87b75924ca030`` **Author:** CrashOverBurn <151276996+crash-over-burn@users.noreply.github.com> ------------------------------------------------------------------------------ Documentation & Website Era (2026-Q2) - CURRENT ------------------------------------------------------------------------------ **The Realization** Code without documentation is incomplete. As nixtamal.tech prepares to launch, comprehensive documentation became essential: **Documentation Sprint:** 1. **Project overview** (doc/index.rst) - Current state summary - Quick links - Architecture overview 2. **Philosophy document** (doc/philosophy.rst) - Why KDL? - VCS diversity rationale - Anti-corporate stance - Privacy principles 3. **Roadmap** (doc/roadmap.rst) - Phased development plan - Phase 3: Visual dependency graphs (next) - Phase 4: Future enhancements 4. **History** (doc/history.rst) - this document - Project origins - Key decisions - Migration stories **Test Coverage:** - bisect_ppx integration - Coverage reporting in CI - Target: 50%+ coverage - New tests: upgrade, Fossil, lockfile **Website Preparation:** - nixtamal.tech domain secured - Static site generation planned - Interactive examples prepared - Community links established ------------------------------------------------------------------------------ Key Decisions & Their Impact ------------------------------------------------------------------------------ **2024: OCaml over Rust/Go** *Decision:* Use OCaml despite Rust's popularity *Impact:* Excellent type safety, maintainable codebase, slower initial development *Status:* ✅ Validated - codebase stable and maintainable **2024: KDL over JSON/YAML** *Decision:* KDL for configuration format *Impact:* Positive user feedback on readability, some tooling gaps *Status:* ✅ Validated - users appreciate the format **2025: Self-hosting over GitHub** *Decision:* Avoid GitHub/Microsoft platforms *Impact:* Reduced visibility, maintained independence *Status:* ✅ Validated - aligns with philosophy **2025-2026: Dual workflow (traditional + flakes)** *Decision:* Support both Nix workflows *Impact:* Broader adoption, more complex codebase *Status:* ✅ Validated - users appreciate choice **2026: Prioritize graphs over TUI** *Decision:* Visual dependency graphs before TUI *Impact:* Web-first approach, SBOM replacement goal *Status:* 🔄 In progress - Phase 3 ------------------------------------------------------------------------------ Community & Contributions ------------------------------------------------------------------------------ **Current State:** - Primary: toastal (author) - XMPP chat: xmpp:nixtamal@chat.toastal.in.th?join - Website: https://nixtamal.tech (launching) - Source: Self-hosted Git repository **Contribution History:** While primarily a solo project, contributions have come from: - Subagent-assisted development (Bellana-Drone, TPol, etc.) - Upstream patches (toastal on Darcs) - OCaml ecosystem improvements (indirect) **Future:** As nixtamal.tech launches, community growth is expected: - Clear contribution guidelines (Phase 2.5) - Issue templates - Beginner-friendly tasks - Sponsorship opportunities ------------------------------------------------------------------------------ Lessons Learned ------------------------------------------------------------------------------ **Technical:** 1. **Type systems are worth it** - OCaml's strict typing caught many bugs 2. **Concurrency is hard** - Eio helped, but still complex 3. **Format stability matters** - Schema versioning was the right call 4. **Test early, test often** - Coverage infrastructure pays off **Social:** 1. **Principles have costs** - Self-hosting reduces visibility 2. **Tooling friction is real** - Darcs → Git migration validated 3. **Documentation is code** - As important as functionality 4. **Community takes time** - Sustainable pace over hype **Project Management:** 1. **Phased development works** - Clear milestones help focus 2. **Upstream tracking is essential** - Port before divergence grows 3. **Automation saves time** - CI/CD, coverage, docs generation 4. **Defer complexity** - TUI, patch management can wait ------------------------------------------------------------------------------ The Future ------------------------------------------------------------------------------ From here, Nixtamal aims to: 1. **Launch nixtamal.tech** with comprehensive documentation 2. **Build visual dependency graphs** (Phase 3) to replace detsys SBOM 3. **Grow community** through clear contribution paths 4. **Maintain principles** while improving accessibility 5. **Become the default** for Nix dependency management The journey from a personal tool to a community project continues. The foundation is solid. The vision is clear. The work continues. ------------------------------------------------------------------------------ .. vim: set textwidth=80