diff options
| author | Crash Over Burn | 2026-04-15 00:48:45 +0000 |
|---|---|---|
| committer | ยท๐๐ด๐๐๐ฉ๐ค | 2026-04-15 00:48:45 +0000 |
| commit | 6e398ebdf0b0bf656a34ed4dd3b92963d684a888 (patch) | |
| tree | 37eeca14e6e41ff23820adcb239f374bf844be35 /doc/history.rst | |
| parent | e6c901d2e5430a3815109b38ced11a4f695f0226 (diff) | |
| download | nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.gz nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.bz2 nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.lz nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.xz nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.tar.zst nixtaml-6e398ebdf0b0bf656a34ed4dd3b92963d684a888.zip | |
Add comprehensive project documentation
Create complete documentation set in doc/ folder:
- index.rst: Main project overview, quickstart, architecture
- philosophy.rst: Design principles, KDL rationale, VCS diversity
- roadmap.rst: Phased development plan (Phase 2.5 current, Phase 3 graphs next)
- history.rst: Development timeline, key decisions, lessons learned
- manifest.rst: Updated to v0.2.0, added Fossil docs, schema upgrade notes
Documentation covers:
- Current state (Phase 2.5)
- Complete roadmap through Phase 4
- Visual dependency graph plans (Phase 3)
- Philosophy and anti-corporate stance
- History from origins to present
All documentation in reStructuredText format for website generation.
Diffstat (limited to 'doc/history.rst')
| -rw-r--r-- | doc/history.rst | 346 |
1 files changed, 346 insertions, 0 deletions
diff --git a/doc/history.rst b/doc/history.rst new file mode 100644 index 0000000..f34741e --- /dev/null +++ b/doc/history.rst @@ -0,0 +1,346 @@ +================================================================================ +Nixtamal Development History +================================================================================ + +.. role:: ab +.. role:: ac + +------------------------------------------------------------------------------ +Origins (2024) +------------------------------------------------------------------------------ + +**The Problem** + +In 2024, toastal was working with Nix in environments that required diverse +version control systems. The Nix ecosystem at the time was heavily Git-centric: + +- Flakes only supported Git repositories well +- Darcs and Pijul projects required manual workarounds +- No standardized way to pin non-Git dependencies +- Existing tools (niv, npins) had limited VCS support + +**The First Spark** + +The initial idea was simple: a version pinning tool that treats all VCS equally. +Not Git-first with other VCS as afterthoughts, but truly first-class support +for the diversity of version control systems. + +**Technology Choice** + +OCaml was chosen for several reasons: +- Strong type system for complex domain modeling +- Excellent concurrency support (Eio library) +- Decades of stability and maintainability +- Personal familiarity +- Nix integration already well-established + +KDL was chosen for configuration because: +- More readable than JSON +- More structured than TOML +- No YAML whitespace hell +- Native comment support (important for hand-edited files) + +------------------------------------------------------------------------------ +Early Development (2024-2025) +------------------------------------------------------------------------------ + +**Darcs-First Approach** + +The project was initially developed in Darcs, hosted at: +``https://darcs.toastal.in.th/nixtamal/trunk`` + +This was a deliberate choice: +- Dogfooding the VCS diversity principle +- Darcs' patch theory is elegant +- Self-hosting independence + +**Version 0.1.x Series** + +The early versions focused on establishing the core: + +- 0.1.0 - Basic manifest/lockfile structure +- 0.1.1 - Stabilized schema with proper versioning +- Git, Darcs, and Pijul support added iteratively +- File and Archive inputs for completeness + +**Key Technical Decisions:** + +1. **Concurrent prefetching** using OCaml 5's Eio library +2. **Template system** using Jingoo for dynamic URLs +3. **KDL codecs** for type-safe configuration parsing +4. **Lockfile format** as JSON (machine-readable, diffable) + +------------------------------------------------------------------------------ +The Nilla Integration (2025) +------------------------------------------------------------------------------ + +**What is Nilla?** + +Nilla (https://github.com/nilla-nix/nilla) is a Nix framework with loaders +and a module system. It represents a new approach to Nix project structure. + +**Integration Work** + +Adding Nilla support required: +- New input type for Nilla references +- Understanding Nilla's loader architecture +- Manifest serialization for Nilla inputs +- Lockfile integration + +**Status:** Types, codecs, and manifest serialization implemented. +Prefetch support deferred pending Nilla ecosystem maturation. + +**Impact:** Nixtamal can now manage Nilla-based inputs alongside traditional +Nix dependencies, supporting hybrid workflows. + +------------------------------------------------------------------------------ +The Git Migration (2025-2026) +------------------------------------------------------------------------------ + +**Why Migrate?** + +Despite philosophical alignment with Darcs, practical realities emerged: + +1. **Contributor friction** - Most developers know Git, not Darcs +2. **Tooling gaps** - CI/CD, GitHub Actions, etc. built around Git +3. **Visibility** - Git repositories get more community attention +4. **Accessibility** - Easier for contributors to submit patches + +**The Fork** + +A community-maintained Git fork was created. The original Darcs repository +continues to exist as the "source of truth" with the Git repository as a +mirror with additional development. + +**Maintaining Principles:** + +The migration didn't compromise core principles: +- Still self-hosted (not GitHub) +- XMPP remains primary communication +- Website still independent +- Philosophy document unchanged + +------------------------------------------------------------------------------ +The Flake Integration (2025-2026) +------------------------------------------------------------------------------ + +**The Controversy** + +Nix flakes are controversial in the community: +- Pro: Integrated locking, caching, composability +- Con: JSON configuration, Git lockfiles, centralized registry + +**The Compromise** + +Rather than choosing sides, Nixtamal embraced both: + +**Traditional Nix** (Default) +- KDL configuration (human-readable) +- External lockfiles +- Works with ``nix-build``, ``nix-shell`` + +**Flake Integration** (Optional) +- ``nixtamal.lib`` for flake consumption +- Hybrid workflows supported +- Use nixtamal projects as flake inputs + +**Technical Implementation:** + +- ``flake.nix`` with proper overlay system +- ``nix/tamal/`` directory for lockfile consumption +- Library functions for ecosystem integration +- Phase 1 implementation notes documented + +**Outcome:** Users choose their workflow. No forced migration. + +------------------------------------------------------------------------------ +Major Upstream Port (2026-04-14) +------------------------------------------------------------------------------ + +**The Gap** + +The Git fork had diverged from upstream Darcs: +- Git fork: 145 commits, ~0.3.x-alpha +- Darcs upstream: 347 patches, v1.1.2 +- ~202 patches behind +- Missing: schema upgrade, Fossil VCS, TUI, etc. + +**The Port** + +A systematic porting effort was undertaken: + +**Phase 1 - Critical Fixes:** +- Cmdliner 2.x compatibility (variable shadowing fixes) +- Lockfile auto-creation (don't error on missing lockfile) + +**Phase 2 - Major Features:** +- Schema upgrade command with backup/rollback +- Fossil VCS support (complete implementation) +- Hash SRI parsing improvements +- Error type refinements + +**Challenges:** + +1. **Type compatibility** - Error types evolved between versions +2. **Module renames** - Foreman vs input_foreman +3. **Pattern match completeness** - Fossil added to all pattern matches +4. **Flake preservation** - Must maintain existing flake integration + +**Resolution:** + +Bellana-Sugar and Bellana-Drone collaborated to resolve type issues while +preserving the Git fork's flake additions. All tests passing. + +**Commit:** +``4ed722f8be5a3bb5ea8ffdfe89f87b75924ca030`` + +**Author:** CrashOverBurn <151276996+crash-over-burn@users.noreply.github.com> + +------------------------------------------------------------------------------ +Documentation & Website Era (2026-Q2) - CURRENT +------------------------------------------------------------------------------ + +**The Realization** + +Code without documentation is incomplete. As nixtamal.tech prepares to launch, +comprehensive documentation became essential: + +**Documentation Sprint:** + +1. **Project overview** (doc/index.rst) + - Current state summary + - Quick links + - Architecture overview + +2. **Philosophy document** (doc/philosophy.rst) + - Why KDL? + - VCS diversity rationale + - Anti-corporate stance + - Privacy principles + +3. **Roadmap** (doc/roadmap.rst) + - Phased development plan + - Phase 3: Visual dependency graphs (next) + - Phase 4: Future enhancements + +4. **History** (doc/history.rst) - this document + - Project origins + - Key decisions + - Migration stories + +**Test Coverage:** + +- bisect_ppx integration +- Coverage reporting in CI +- Target: 50%+ coverage +- New tests: upgrade, Fossil, lockfile + +**Website Preparation:** + +- nixtamal.tech domain secured +- Static site generation planned +- Interactive examples prepared +- Community links established + +------------------------------------------------------------------------------ +Key Decisions & Their Impact +------------------------------------------------------------------------------ + +**2024: OCaml over Rust/Go** + +*Decision:* Use OCaml despite Rust's popularity +*Impact:* Excellent type safety, maintainable codebase, slower initial development +*Status:* โ
Validated - codebase stable and maintainable + +**2024: KDL over JSON/YAML** + +*Decision:* KDL for configuration format +*Impact:* Positive user feedback on readability, some tooling gaps +*Status:* โ
Validated - users appreciate the format + +**2025: Self-hosting over GitHub** + +*Decision:* Avoid GitHub/Microsoft platforms +*Impact:* Reduced visibility, maintained independence +*Status:* โ
Validated - aligns with philosophy + +**2025-2026: Dual workflow (traditional + flakes)** + +*Decision:* Support both Nix workflows +*Impact:* Broader adoption, more complex codebase +*Status:* โ
Validated - users appreciate choice + +**2026: Prioritize graphs over TUI** + +*Decision:* Visual dependency graphs before TUI +*Impact:* Web-first approach, SBOM replacement goal +*Status:* ๐ In progress - Phase 3 + +------------------------------------------------------------------------------ +Community & Contributions +------------------------------------------------------------------------------ + +**Current State:** + +- Primary: toastal (author) +- XMPP chat: xmpp:nixtamal@chat.toastal.in.th?join +- Website: https://nixtamal.tech (launching) +- Source: Self-hosted Git repository + +**Contribution History:** + +While primarily a solo project, contributions have come from: +- Subagent-assisted development (Bellana-Drone, TPol, etc.) +- Upstream patches (toastal on Darcs) +- OCaml ecosystem improvements (indirect) + +**Future:** + +As nixtamal.tech launches, community growth is expected: +- Clear contribution guidelines (Phase 2.5) +- Issue templates +- Beginner-friendly tasks +- Sponsorship opportunities + +------------------------------------------------------------------------------ +Lessons Learned +------------------------------------------------------------------------------ + +**Technical:** + +1. **Type systems are worth it** - OCaml's strict typing caught many bugs +2. **Concurrency is hard** - Eio helped, but still complex +3. **Format stability matters** - Schema versioning was the right call +4. **Test early, test often** - Coverage infrastructure pays off + +**Social:** + +1. **Principles have costs** - Self-hosting reduces visibility +2. **Tooling friction is real** - Darcs โ Git migration validated +3. **Documentation is code** - As important as functionality +4. **Community takes time** - Sustainable pace over hype + +**Project Management:** + +1. **Phased development works** - Clear milestones help focus +2. **Upstream tracking is essential** - Port before divergence grows +3. **Automation saves time** - CI/CD, coverage, docs generation +4. **Defer complexity** - TUI, patch management can wait + +------------------------------------------------------------------------------ +The Future +------------------------------------------------------------------------------ + +From here, Nixtamal aims to: + +1. **Launch nixtamal.tech** with comprehensive documentation +2. **Build visual dependency graphs** (Phase 3) to replace detsys SBOM +3. **Grow community** through clear contribution paths +4. **Maintain principles** while improving accessibility +5. **Become the default** for Nix dependency management + +The journey from a personal tool to a community project continues. The +foundation is solid. The vision is clear. The work continues. + +------------------------------------------------------------------------------ +.. vim: set textwidth=80 |
