The past year has seen a surge in interest in Postgres extension distribution. A number of people have noted in particular the challenges in finding and installing extensions.
PGXN, released way back in 2011, aimed to be the canonical registry for the community, but currently indexes only the 350 distributions that developers have made the effort to publish — perhaps a third of all known extensions. Furthermore, PGXN distributes source code packages, requiring developers to download, compile, install, and test them.
As a result of these challenges, a number of new entrants have emerged in recent months with a focus on discovery and ease of installation. These include dbdev, pgxman, pgpm, and our own trunk. However, they too exhibit limitations, such as minimal OS and platform availability and, notably, the manual process to add extensions, further constraining inclusion. None has even reached the number provided by PGXN.
These challenges and the interest and energy put into exploring new solutions make clear that the time has come to revisit the whole idea of the PostgreSQL extension ecosystem: to work though the jobs to be done, specify the tools to do those jobs, and outline a plan for the broader Postgres community to design and build them.
Future posts will dream up the tools and make the plan; today we begin with the jobs.
🎬 Let’s get started.
Jobs to be Done
Theodore Levitt cites a famous adage:
“Last year 1 million quarter-inch drills were sold,” Leo McGivena once said, “not because people wanted quarter-inch drills but because they wanted quarter-inch holes.”
People don’t buy products; they buy the expectation of benefits.
Today the most cited descriptor underpinning Jobs to be Done Theory, thanks to Clayton Christensen, we bear it in mind to think through the “jobs” of an idealized Postgres extension ecosystem — without reference to existing Postgres solutions. However, we cite examples for the jobs from other projects and communities, both to clarify the jobs to be done and to reference prior art from which to borrow.
So what are the jobs to be done by the ideal Postgres extension ecosystem?
Authoritative
Job: Be the official, canonical source of record for all Postgres extensions
Developers need to know the one place to reliably release and find all publicly-available Postgres extensions. It can’t be a subset, but must be comprehensive, the assumed default recognized and used by the community and included in the Postgres documentation.
Exemplars:
- The Go Module Index indexes Go package that it automatically discovers
- crates.io indexes Rust crates published to it
- PAUSE indexes Perl modules uploaded to it
- RubyGems indexes Ruby Gems published to it
- PyPI indexes Python distributions uploaded to it
Beneficial
Job: It’s expected for extension developers to develop and publish their extensions in the ecosystem.
Extension developers want their extensions to be a part of the ecosystem, to gain all the advantages and privileges of participation, without undo effort.
Publishing an extension should be as straightforward as possible. Ideally it simply requires maintaining a metadata file and publishing releases to the authoritative index.
Exemplars:
- Go Package Discovery automatically discovers and indexes
go.mod-configured packages - Registered crates.io developers publish Rust crates defined by Cargo
Manifest Files via the
cargoCLI - Registered NPM developers publish NPM projects defined by
package.jsonfiles via thenpm publishcommand - Registered PAUSE developers publish Perl modules defined by CPAN Meta
Spec files via the
cpan-uploadCLI or web API
Integrated
Job: It’s easy for developers to start writing and publishing extensions.
Extension developers need excellent tooling to simplify the process of creating, maintaining, testing, and distributing extensions. Such tools should be well-integrated into the extension ecosystem, and empower developers to get started and easily follow best practices throughout the extension development lifecycle: from beginning development, managing dependencies and other metadata, testing it, and releasing it.
Exemplars:
Discoverable
Job: Make extensions easy to find and understand
There’s no use in being the canonical index of extensions if no one can find them. The ecosystem needs a well-known web site where people find extensions, learn their purposes, distinguish them from each other, and read their docs. Ideally most people use integrated search rather than a search engine, but extension pages should appear high up in search engine results, too.
Exemplars:
Informative
Job: Make extension documentation consistent, standardized, and comprehensive
Extensions are only as easy to use as their documentation enables. It’s therefore important to provide well-organized, standardized, and accessible documentation — and that it be expedient for developers to document their extension packages. Ideally development tools encourage extension documentation. A documentation standard enables publishing docs in multiple formats, especially on the canonical discovery site.
Exemplars:
Programmable
Job: Provide stable, easy-to-use, and comprehensive APIs to empower downstream developers to build and integrate other tools and products
Provide excellent, well-documented, intuitive web APIs that allow anyone to build useful tools and services on the index. The jobs below require it, but creative coders will invent and mashup APIs in ways we haven’t thought of. Web APIs enable imaginative solutions within the ecosystem and without. Some example APIs:
- Recent releases feeds
- Extension metadata
- Full text search
- Event webhooks to notify registered services
Better still, let trusted services report back results for listing — even badging! — in the authoritative index, such as download counts, build results, test results, binary packaging availability, and more.
Exemplars:
Installable
Job: Provide automatic binary packaging for wide variety of platforms
It must be as easy as possible for users to install all indexed extensions. Ideally, independent services rely on the index APIs for notification of new releases and automatically build binary packages for as wide a variety of OSes, OS versions, architectures (arm64, amd64, etc.), and PostgreSQL versions as possible.
Some example repositories to support or enable to build and distribute extensions from the core registry:
- Community packaging: apt.postgresql.org, yum.postgresql.org
- Binary packaging: Homebrew, Chocolatey, etc.
- Vendor packaging & distribution: pgxman, dbdev, pgpm, trunk
Exemplars: ❓❓
Trusted
Job: Validate extensions and protect from supply chain vulnerabilities
Provide tools and features to earn and retain user trust by ensuring that that extensions indexed by the canonical registry have been validated, come from trustworthy sources, and are unlikely to be a vector for supply chain attacks. This means some combination of author validation (e.g. badging projects from well-known sources), checksums, public key signing, and perhaps static code analysis.
Exemplars:
- How Go Mitigates Supply Chain Attacks
- Improving Supply Chain Security for Rust Through Artifact Signing
- GitHub Code Scanning
- Docker Scout
Manageable
Job: Provide intuitive, powerful interfaces for installing and managing extensions
Provide applications to manage the extensions in one or more Postgres clusters. At a minimum distribute a command line interface that manages extensions for a locally-installed Postgres — as currently required for it to install packages, for example.
Users need to easily manage their Postgres extensions in a single, consistent manner. The extensions manager should be aware of installed extensions, available extensions, dependency graphs, and known vulnerabilities; allow for search (via the APIs); and of course install or upgrade extensions.
Exemplars:
Omissions and Next Steps
With these jobs in mind, we begin to identify the tools necessary to serve them in the next post.
But first, what’s missing here? Are these nine jobs to be done sufficiently comprehensive to meet most extension indexing, discovery, packaging, and distribution needs? What’s missing? We don’t want to leave any good ideas behind! Please drop yours into the #extensions channel on the Postgres Slack.
