summaryrefslogtreecommitdiff
path: root/doc/history.rst
diff options
context:
space:
mode:
authorCrash Over Burn2026-04-15 00:48:45 +0000
committerยท๐‘‘๐‘ด๐‘•๐‘‘๐‘ฉ๐‘ค2026-04-15 00:48:45 +0000
commit6e398ebdf0b0bf656a34ed4dd3b92963d684a888 (patch)
tree37eeca14e6e41ff23820adcb239f374bf844be35 /doc/history.rst
parente6c901d2e5430a3815109b38ced11a4f695f0226 (diff)
downloadnixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar
nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.gz
nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.bz2
nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.lz
nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.xz
nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.zst
nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.zip
Add comprehensive project documentation
Create complete documentation set in doc/ folder: - index.rst: Main project overview, quickstart, architecture - philosophy.rst: Design principles, KDL rationale, VCS diversity - roadmap.rst: Phased development plan (Phase 2.5 current, Phase 3 graphs next) - history.rst: Development timeline, key decisions, lessons learned - manifest.rst: Updated to v0.2.0, added Fossil docs, schema upgrade notes Documentation covers: - Current state (Phase 2.5) - Complete roadmap through Phase 4 - Visual dependency graph plans (Phase 3) - Philosophy and anti-corporate stance - History from origins to present All documentation in reStructuredText format for website generation.
Diffstat (limited to 'doc/history.rst')
-rw-r--r--doc/history.rst346
1 files changed, 346 insertions, 0 deletions
diff --git a/doc/history.rst b/doc/history.rst
new file mode 100644
index 0000000..f34741e
--- /dev/null
+++ b/doc/history.rst
@@ -0,0 +1,346 @@
+================================================================================
+Nixtamal Development History
+================================================================================
+
+.. role:: ab
+.. role:: ac
+
+------------------------------------------------------------------------------
+Origins (2024)
+------------------------------------------------------------------------------
+
+**The Problem**
+
+In 2024, toastal was working with Nix in environments that required diverse
+version control systems. The Nix ecosystem at the time was heavily Git-centric:
+
+- Flakes only supported Git repositories well
+- Darcs and Pijul projects required manual workarounds
+- No standardized way to pin non-Git dependencies
+- Existing tools (niv, npins) had limited VCS support
+
+**The First Spark**
+
+The initial idea was simple: a version pinning tool that treats all VCS equally.
+Not Git-first with other VCS as afterthoughts, but truly first-class support
+for the diversity of version control systems.
+
+**Technology Choice**
+
+OCaml was chosen for several reasons:
+- Strong type system for complex domain modeling
+- Excellent concurrency support (Eio library)
+- Decades of stability and maintainability
+- Personal familiarity
+- Nix integration already well-established
+
+KDL was chosen for configuration because:
+- More readable than JSON
+- More structured than TOML
+- No YAML whitespace hell
+- Native comment support (important for hand-edited files)
+
+------------------------------------------------------------------------------
+Early Development (2024-2025)
+------------------------------------------------------------------------------
+
+**Darcs-First Approach**
+
+The project was initially developed in Darcs, hosted at:
+``https://darcs.toastal.in.th/nixtamal/trunk``
+
+This was a deliberate choice:
+- Dogfooding the VCS diversity principle
+- Darcs' patch theory is elegant
+- Self-hosting independence
+
+**Version 0.1.x Series**
+
+The early versions focused on establishing the core:
+
+- 0.1.0 - Basic manifest/lockfile structure
+- 0.1.1 - Stabilized schema with proper versioning
+- Git, Darcs, and Pijul support added iteratively
+- File and Archive inputs for completeness
+
+**Key Technical Decisions:**
+
+1. **Concurrent prefetching** using OCaml 5's Eio library
+2. **Template system** using Jingoo for dynamic URLs
+3. **KDL codecs** for type-safe configuration parsing
+4. **Lockfile format** as JSON (machine-readable, diffable)
+
+------------------------------------------------------------------------------
+The Nilla Integration (2025)
+------------------------------------------------------------------------------
+
+**What is Nilla?**
+
+Nilla (https://github.com/nilla-nix/nilla) is a Nix framework with loaders
+and a module system. It represents a new approach to Nix project structure.
+
+**Integration Work**
+
+Adding Nilla support required:
+- New input type for Nilla references
+- Understanding Nilla's loader architecture
+- Manifest serialization for Nilla inputs
+- Lockfile integration
+
+**Status:** Types, codecs, and manifest serialization implemented.
+Prefetch support deferred pending Nilla ecosystem maturation.
+
+**Impact:** Nixtamal can now manage Nilla-based inputs alongside traditional
+Nix dependencies, supporting hybrid workflows.
+
+------------------------------------------------------------------------------
+The Git Migration (2025-2026)
+------------------------------------------------------------------------------
+
+**Why Migrate?**
+
+Despite philosophical alignment with Darcs, practical realities emerged:
+
+1. **Contributor friction** - Most developers know Git, not Darcs
+2. **Tooling gaps** - CI/CD, GitHub Actions, etc. built around Git
+3. **Visibility** - Git repositories get more community attention
+4. **Accessibility** - Easier for contributors to submit patches
+
+**The Fork**
+
+A community-maintained Git fork was created. The original Darcs repository
+continues to exist as the "source of truth" with the Git repository as a
+mirror with additional development.
+
+**Maintaining Principles:**
+
+The migration didn't compromise core principles:
+- Still self-hosted (not GitHub)
+- XMPP remains primary communication
+- Website still independent
+- Philosophy document unchanged
+
+------------------------------------------------------------------------------
+The Flake Integration (2025-2026)
+------------------------------------------------------------------------------
+
+**The Controversy**
+
+Nix flakes are controversial in the community:
+- Pro: Integrated locking, caching, composability
+- Con: JSON configuration, Git lockfiles, centralized registry
+
+**The Compromise**
+
+Rather than choosing sides, Nixtamal embraced both:
+
+**Traditional Nix** (Default)
+- KDL configuration (human-readable)
+- External lockfiles
+- Works with ``nix-build``, ``nix-shell``
+
+**Flake Integration** (Optional)
+- ``nixtamal.lib`` for flake consumption
+- Hybrid workflows supported
+- Use nixtamal projects as flake inputs
+
+**Technical Implementation:**
+
+- ``flake.nix`` with proper overlay system
+- ``nix/tamal/`` directory for lockfile consumption
+- Library functions for ecosystem integration
+- Phase 1 implementation notes documented
+
+**Outcome:** Users choose their workflow. No forced migration.
+
+------------------------------------------------------------------------------
+Major Upstream Port (2026-04-14)
+------------------------------------------------------------------------------
+
+**The Gap**
+
+The Git fork had diverged from upstream Darcs:
+- Git fork: 145 commits, ~0.3.x-alpha
+- Darcs upstream: 347 patches, v1.1.2
+- ~202 patches behind
+- Missing: schema upgrade, Fossil VCS, TUI, etc.
+
+**The Port**
+
+A systematic porting effort was undertaken:
+
+**Phase 1 - Critical Fixes:**
+- Cmdliner 2.x compatibility (variable shadowing fixes)
+- Lockfile auto-creation (don't error on missing lockfile)
+
+**Phase 2 - Major Features:**
+- Schema upgrade command with backup/rollback
+- Fossil VCS support (complete implementation)
+- Hash SRI parsing improvements
+- Error type refinements
+
+**Challenges:**
+
+1. **Type compatibility** - Error types evolved between versions
+2. **Module renames** - Foreman vs input_foreman
+3. **Pattern match completeness** - Fossil added to all pattern matches
+4. **Flake preservation** - Must maintain existing flake integration
+
+**Resolution:**
+
+Bellana-Sugar and Bellana-Drone collaborated to resolve type issues while
+preserving the Git fork's flake additions. All tests passing.
+
+**Commit:**
+``4ed722f8be5a3bb5ea8ffdfe89f87b75924ca030``
+
+**Author:** CrashOverBurn <151276996+crash-over-burn@users.noreply.github.com>
+
+------------------------------------------------------------------------------
+Documentation & Website Era (2026-Q2) - CURRENT
+------------------------------------------------------------------------------
+
+**The Realization**
+
+Code without documentation is incomplete. As nixtamal.tech prepares to launch,
+comprehensive documentation became essential:
+
+**Documentation Sprint:**
+
+1. **Project overview** (doc/index.rst)
+ - Current state summary
+ - Quick links
+ - Architecture overview
+
+2. **Philosophy document** (doc/philosophy.rst)
+ - Why KDL?
+ - VCS diversity rationale
+ - Anti-corporate stance
+ - Privacy principles
+
+3. **Roadmap** (doc/roadmap.rst)
+ - Phased development plan
+ - Phase 3: Visual dependency graphs (next)
+ - Phase 4: Future enhancements
+
+4. **History** (doc/history.rst) - this document
+ - Project origins
+ - Key decisions
+ - Migration stories
+
+**Test Coverage:**
+
+- bisect_ppx integration
+- Coverage reporting in CI
+- Target: 50%+ coverage
+- New tests: upgrade, Fossil, lockfile
+
+**Website Preparation:**
+
+- nixtamal.tech domain secured
+- Static site generation planned
+- Interactive examples prepared
+- Community links established
+
+------------------------------------------------------------------------------
+Key Decisions & Their Impact
+------------------------------------------------------------------------------
+
+**2024: OCaml over Rust/Go**
+
+*Decision:* Use OCaml despite Rust's popularity
+*Impact:* Excellent type safety, maintainable codebase, slower initial development
+*Status:* โœ… Validated - codebase stable and maintainable
+
+**2024: KDL over JSON/YAML**
+
+*Decision:* KDL for configuration format
+*Impact:* Positive user feedback on readability, some tooling gaps
+*Status:* โœ… Validated - users appreciate the format
+
+**2025: Self-hosting over GitHub**
+
+*Decision:* Avoid GitHub/Microsoft platforms
+*Impact:* Reduced visibility, maintained independence
+*Status:* โœ… Validated - aligns with philosophy
+
+**2025-2026: Dual workflow (traditional + flakes)**
+
+*Decision:* Support both Nix workflows
+*Impact:* Broader adoption, more complex codebase
+*Status:* โœ… Validated - users appreciate choice
+
+**2026: Prioritize graphs over TUI**
+
+*Decision:* Visual dependency graphs before TUI
+*Impact:* Web-first approach, SBOM replacement goal
+*Status:* ๐Ÿ”„ In progress - Phase 3
+
+------------------------------------------------------------------------------
+Community & Contributions
+------------------------------------------------------------------------------
+
+**Current State:**
+
+- Primary: toastal (author)
+- XMPP chat: xmpp:nixtamal@chat.toastal.in.th?join
+- Website: https://nixtamal.tech (launching)
+- Source: Self-hosted Git repository
+
+**Contribution History:**
+
+While primarily a solo project, contributions have come from:
+- Subagent-assisted development (Bellana-Drone, TPol, etc.)
+- Upstream patches (toastal on Darcs)
+- OCaml ecosystem improvements (indirect)
+
+**Future:**
+
+As nixtamal.tech launches, community growth is expected:
+- Clear contribution guidelines (Phase 2.5)
+- Issue templates
+- Beginner-friendly tasks
+- Sponsorship opportunities
+
+------------------------------------------------------------------------------
+Lessons Learned
+------------------------------------------------------------------------------
+
+**Technical:**
+
+1. **Type systems are worth it** - OCaml's strict typing caught many bugs
+2. **Concurrency is hard** - Eio helped, but still complex
+3. **Format stability matters** - Schema versioning was the right call
+4. **Test early, test often** - Coverage infrastructure pays off
+
+**Social:**
+
+1. **Principles have costs** - Self-hosting reduces visibility
+2. **Tooling friction is real** - Darcs โ†’ Git migration validated
+3. **Documentation is code** - As important as functionality
+4. **Community takes time** - Sustainable pace over hype
+
+**Project Management:**
+
+1. **Phased development works** - Clear milestones help focus
+2. **Upstream tracking is essential** - Port before divergence grows
+3. **Automation saves time** - CI/CD, coverage, docs generation
+4. **Defer complexity** - TUI, patch management can wait
+
+------------------------------------------------------------------------------
+The Future
+------------------------------------------------------------------------------
+
+From here, Nixtamal aims to:
+
+1. **Launch nixtamal.tech** with comprehensive documentation
+2. **Build visual dependency graphs** (Phase 3) to replace detsys SBOM
+3. **Grow community** through clear contribution paths
+4. **Maintain principles** while improving accessibility
+5. **Become the default** for Nix dependency management
+
+The journey from a personal tool to a community project continues. The
+foundation is solid. The vision is clear. The work continues.
+
+------------------------------------------------------------------------------
+.. vim: set textwidth=80