Skip to main content

5 posts tagged with "announcement"

View All Tags

· 3 min read

tembo-terraform-provider

At Tembo, we want to empower developers to build fast. That’s why we built the Terraform Provider for Tembo Cloud, allowing developers to manage Postgres resources using Infrastructure-as-Code (IaC). In this blog, we'll explore the Terraform Provider for Tembo and how it can help you streamline database management while keeping teams agile, accountable and enforcing organizational policies.

Simpler, faster, safer, and easier? That’s Tembo - but let’s back up and start at the beginning.

What is Tembo Cloud?

Tembo Cloud is a developer-first, fully-extensible, fully-managed, secure, and scalable Postgres service. It not only simplifies Postgres management but also provides specialized Postgres deployments through Tembo Stacks, catering to various use cases.

Database as Code using Terraform

Databases as code provides a safe, consistent, and repeatable way of managing databases and is a game-changer for developers. This approach allows you to version-control any changes, ensuring auditability, and facilitates efficient workflows such as Pull Request flows and GitOps.

Terraform is the most popular Infrastructure as Code tool today. It provides a way to define the desired state of your infrastructure using Hashicorp Configuration Language(HCL). Terraform handles all the complexity of making sure that when changes are applied, the actual state matches the desired state. Therefore, it is an efficient way of managing databases as code.

Using Terraform Provider for Tembo

With the Terraform Provider for Tembo, you can manage Postgres databases on Tembo Cloud with all the benefits mentioned in the previous section. You can find the Terraform Provider for Tembo on the Terraform Registry along with the documentation, and the Github repo is here. A new Postgres instance can be provisioned on Tembo Cloud in about 1 minute and destroying an instance takes just 10 seconds.

Example Terraform file

You can create a new Tembo instance on Tembo Cloud using the tembo_instance resource available with the provider. Below is an example configuration.

Note: Tembo Terraform Provider needs an access_token to authenticate with the API. Generate a long-lived API token by following steps here.

terraform {
required_providers {
tembo = {
source = "tembo-io/tembo"
version = ">= 0.1.0"
}
}
}

provider "tembo" {
access_token = var.access_token
}

variable "access_token" {
type = string
}

resource "tembo_instance" "test_instance" {
instance_name = "test-instance"
org_id = "org_test" # Replace this with your Tembo organization id
cpu = "1"
stack_type = "Standard"
environment = "dev"
memory = "4Gi"
storage = "10Gi"
replicas = 1
extensions = [{
name = "plperl"
description = "PL/Perl procedural language"
locations = [{
database = "app"
schema = "public"
version = "1.0"
enabled = false
},
{
database = "postgres"
schema = "public"
version = "1.0"
enabled = true
}]
}]
}

output "instance" {
value = tembo_instance.test_instance
}

Applying the above Terraform will provision a Postgres Instance on Tembo Cloud. For a demo explaining the steps watch the video below.

Demo

The demo in the video explains how to use the Terraform Provider for Tembo. The Terraform code used in the video can be found here.

What’s next for the Provider

We are actively working on an Import feature that will allow you to import existing Tembo instances to Terraform to easily start managing existing instances with Terraform. Stay updated by starring our GitHub repository and monitoring changes on the Terraform Registry. Most importantly, please try it out, and file any issues and feature requests in our repository. You can also access the documentation for the Tembo provider here.

· 5 min read
Adam Hendel

We’re working on asynchronous query execution in Postgres and have packaged the work up in an extension we’re calling pg_later. If you’ve used Snowflake’s asynchronous queries, then you might already be familiar with this type of feature. Submit your queries to Postgres now, and come back later and get the query’s results.

Visit pg_later’s Github repository and give it a star!

elephant-tasker

Why async queries?

Imagine that you’ve initiated a long-running maintenance job. You step away while it is executing, only to come back and discover it was interrupted hours ago due to your laptop shutting down. You don’t want this to happen again, so you spend some time googling or asking your favorite LLM how to run the command in the background with screen or tmux. Having asynchronous query support from the beginning would have saved you a bunch of time and headache!

Asynchronous processing is a useful development pattern in software engineering. It has advantages such as improved resource utilization, and unblocking of the main execution thread.

Some examples where async querying can be useful are:

  • For a DBA running ad-hoc maintenance.
  • Developing in interactive environments such as a Jupyter notebook. Rather than submit a long-running query only to have your notebook hang idly and then crash, you can use asynchronous tasks to avoid blocking your Notebook, or simply come back later to check on your task.
  • Having a long-running analytical query, for example fulfilling an ad-hoc request like seeing how many new users signed up each day for the past month. You can submit that query and have it run in the background while you continue other work.

Extending Postgres with async features

At Tembo, we’ve built a similar feature for Postgres and published it as an extension called pg_later. With pg_later, you can dispatch a query to your Postgres database and, rather than waiting for the results, your program can return and retrieve the results at your convenience.

A common example is manually executing VACUUM on a table. Typically one might execute VACUUM in one session, and then use another session to check the status of the VACUUM job via pg_stat_progress_vacuum. pg_later gives you the power to do that in a single session. You can use it to queue up any long-running analytical or administrative task on your Postgres database.

Stacking Postgres Extensions

pg_later is built on top of PGMQ, another one of Tembo's open source extensions. Once a user submits a query, pg_later seamlessly enqueues the request in a Postgres-managed message queue. This mechanism then processes the query asynchronously, ensuring no unnecessary wait times or hold-ups.

The pg_later background worker picks up the query from the queue and executes it. The results are persisted by being written to a table as JSONB and can be easily retrieved using the pg_later API. You can simply reference the unique job id given upon query submission, and retrieve the result set, or query the table directly. By default, the results are retained forever, however we are building retention policies as a feature into pg_later.

diagram

Using pg_later

To get started, check out our project’s README for a guide on installing the extension.

First, you need to initialize the extension. This handles the management of PGMQ objects like a job queue and some metadata tables.

select pglater.init();

You're now set to dispatch your queries. Submit the query using pglater.exec, and be sure to take note of the job_id that is returned. In this case, it’s the first job so the job_id is 1.

select pglater.exec(
'select * from pg_available_extensions limit 2'
) as job_id;
 job_id 
--------
1
(1 row)

And whenever you're ready, your results are a query away:

select pglater.fetch_results(1);
{
"query": "select * from pg_available_extensions limit 2",
"job_id": 1,
"result": [
{
"name": "pg_later",
"comment": "pg_later: Run queries now and get results later",
"default_version": "0.0.6",
"installed_version": "0.0.6"
},
{
"name": "pgmq",
"comment": "Distributed message queues",
"default_version": "0.10.1",
"installed_version": "0.10.1"
}
],
"status": "success"
}

Up next

pg_later is a new project and still under development. A few features that we are excited to build:

  • Status and progress of in-flight queries
  • Security and permission models for submitted queries
  • Cursor support for finished jobs (fetch results row by row)
  • Kill a query that is in the queue or is currently in flight
  • Support for transactions
  • Configurable concurrency levels for background works to increase the throughput of jobs
  • Push notifications for completed and failed jobs
  • Retention policies for completed jobs

Give us a star and try out pg_later by running the example in the README. If you run into issues, please create an issue. We would greatly welcome contributions to the project as well.

Please stay tuned for a follow up post on benchmarking PGMQ vs leading open source message queues.

· 5 min read
Adam Hendel

We’ve released PGMQ, a packaged extension for message queues on Postgres.

People have been implementing queues on Postgres in many different ways and we’re excited about combining lessons learned from those projects into a simple, feature-rich extension.

Some exciting features of the project include:

  • Guaranteed exactly-once delivery of messages within a visibility timeout
  • Optional archival of messages retention for replayability and retention
  • Familiar SQL interface
  • Single and Batch read of messages
  • Client SDKs in both Rust and Python for an ORM-like feel

The need for message queues

Message queues are a very common architectural feature to manage operational pipelines, particularly within the context of asynchronous tasks and distributed systems. There are products in the market that support message queues (Kafka, RSMQ, RabbitMQ, SQS); however, when adopting one of these technologies, you increase your cognitive load, required skills and production support overhead.

Building your queues on Postgres

As a Postgres startup, we had the same issue, and we decided to build our own Message Queue based on Postgres. We are not the first: others have implemented queues on Postgres, and many have written about it including Dagster, CrunchyData, and PostgresFM dedicated an entire podcast episode to them.

At Tembo, we needed a job queue to manage tasks between our control-plane and data-plane in our managed cloud offering. Our control-plane publishes tasks like create postgres cluster, and update postgres cluster.

To keep our architecture simple and reduce technology sprawl, we built a Postgres extension so that we could run queues for our cloud and more easily share the implementation with the community.

Queues Implemented with best practices

PGMQ was implemented on Postgres and follows industry best practices. One of the most important practices is the use of Postgres’s SKIP LOCKED, which is similar to NOWAIT in other databases. SKIP LOCKED helps ensure that consumers don't hang and FOR UPDATE ensures messages are not duplicated on read. PGMQ also supports partitioning, which is particularly beneficial for large queues and can be used to efficiently archive / expire old messages.

PGMQ also provides exactly once delivery semantics within a visibility timeout. Similar to Amazon’s SQS and RSMQ, PGMQ consumers set the period of time during which Postgres will prevent all consumers from receiving and processing a message. This is done by the consumer on read, and once the visibility timeout expires the message becomes available for consumption once again. That way, if a consumer crashes, there is no data loss. This effectively means at-least-once delivery semantics once the first visibility timeout has expired.

vt

Using PGMQ

To get started, check out our project’s README for a guide on installing the extension.

You can create a new queue by simply calling

SELECT pgmq_create('my_queue');

Then, pgmq_send() a couple messages to the queue. The message id is returned from the send() function.

SELECT * from pgmq_send('my_queue', '{"foo": "bar1"}');
SELECT * from pgmq_send('my_queue', '{"foo": "bar2"}');
 pgmq_send
-----------
1
(1 row)

pgmq_send
-----------
2
(1 row)

Read 2 messages from the queue. Make them invisible for 30 seconds. If the messages are not deleted or archived within 30 seconds, they will become visible again and can be read by another consumer.

SELECT * from pgmq_read('my_queue', 30, 2);
 msg_id | read_ct |              vt               |          enqueued_at          |    message
--------+---------+-------------------------------+-------------------------------+---------------
1 | 1 | 2023-02-07 04:56:00.650342-06 | 2023-02-07 04:54:51.530818-06 | {"foo":"bar1"}
2 | 1 | 2023-02-07 04:56:00.650342-06 | 2023-02-07 04:54:51.530818-06 | {"foo":"bar2"}

If the queue is empty, or if all messages are currently invisible, no rows will be returned.

SELECT * from pgmq_read('my_queue', 30, 1);
 msg_id | read_ct | vt | enqueued_at | message
--------+---------+----+-------------+---------

Archiving removes the message from the queue and inserts it to the queue’s archive table. This provides you with an opt-in retention mechanism for messages, and is an excellent way to debug applications.

Archive the message with id 2.

SELECT * from pgmq_archive('my_queue', 2);

Then inspect the message on the archive table.

SELECT * from pgmq_my_queue_archive;
 msg_id | read_ct |         enqueued_at          |          deleted_at           |              vt               |     message     
--------+---------+------------------------------+-------------------------------+-------------------------------+-----------------
2 | 1 | 2023-04-25 00:55:40.68417-05 | 2023-04-25 00:56:35.937594-05 | 2023-04-25 00:56:20.532012-05 | {"foo": "bar2"}```

Alternatively, you can delete a message forever.

SELECT * from pgmq_send('my_queue', '{"foo": "bar3"}');
 pgmq_send
-----------
3
(1 row)
SELECT pgmq_delete('my_queue', 3);
 pgmq_delete
-------------
t

Getting involved

Give us a star and try out PGMQ by cloning the repo and following the example in the README. Please use Github issues if you run into any issues or have any feedback.

We’ve also built client side libraries in Rust and Python, which will give you an ORM-like experience.

You can also try PGMQ on Tembo Cloud as part of our Message Queue Stack. Tembo Cloud’s Message Queue Stack is powered by PGMQ, but also ships with Postgres configurations optimized for message queue workloads. We’re also working on adding metrics and data visualizations specific to message queues.

Interested in learning more?

Stay tuned for our upcoming post pg_later, an extension we built on top of PGMQ as well as benchmarks comparing PGMQ to SQS and Redis.

· 6 min read
Ry Walker

tembo brand

$100 billion+

That's the expected size of the global database market in the coming years. The amount of data being generated, stored, and leveraged is growing exponentially, as is the need for applications to operate at a global scale. We're producing more data, we need it faster, and the uses for it are more and more complex by the day.

There's an incredible opportunity in the making here. But first, a little context.

The Database Market

Enterprises typically store their data across various databases, generally grouped into transactional and analytical systems. There's roughly 10x more transactional than analytical data, mostly in Oracle, MySQL, and Postgres, and there have been some sizable shifts to this landscape in recent years:

  1. First, we saw some analytical workloads move to the cloud. This was the big data era, resulting in the rise of platforms like Snowflake and Databricks.
  2. Next, some transactional workloads moved to streaming and real-time data. This required transactional and analytical processing platforms to be managed by app developers, not database admins.
  3. Finally, application infrastructure has been abstracted, allowing developers to build and scale applications more efficiently. Services like Vercel and Netlify have streamlined the development lifecycle, but tend to build on top of databases rather than dealing with the databases themselves.

The net result? Exactly what we see around us—an ever-expanding menagerie of innovative apps and services built upon an increasingly complex database ecosystem. The modern data stack is more complicated and expensive than it's ever been, and with the normalization of AI, the trend is accelerating.

How can we cope?

Postgres as a Platform

Developers would love an open-source, multi-functional data platform to simplify their lives and work—as long as it doesn't restrict their freedom. Companies want reliability, flexibility, and a pathway out of the spiraling costs and vendor lock-in attempts that they're currently saddled with. At face value, there's an obvious solution that ticks all the necessary boxes:

Postgres.

Postgres, the world's favorite database with millions of deployments, features a liberal OSS license and a large community. It efficiently manages SQL and JSON queries across diverse workloads due to its growing, decade-old ecosystem of add-ons and extensions. It is popular for its open-source, standards-compliant, extensible nature, and ACID compliance, making it a reliable, cost-effective system. It handles low latency, high throughput analytical cases, offering HTAP-lite capabilities through window functions and foreign data wrappers. Even better, it’s extensibility has resulted in a wide ecosystem of add-ons and plugins for GIS data, image processing, vector databases, and more, with some extensions evolving into companies like CitusDB and Timescale. In short, everything you’d want and then some.

Postgres is most admired and desired

Source: Stack Overflow Developer Survey 2023

Problem solved, right? Not so fast.

Overwhelmment

(It's a word now. Go with it.)

Companies are typically hesitant to adopt databases due to costs, complexity, and risk. The effort required to build, configure, and optimize a new system often makes the transition value negligible at best. For these companies (especially large enterprises that spend billions per year on a fragmented database architecture), Postgres and it's assimilation of database innovations should be an ideal solution. Open source, extensible, free from vendor lock in and ever-increasing costs—it should be a no brainer.

“What's the holdup?” you might say. Well, Postgres is... complicated.

To create a self-managed cluster of Postgres clusters, DBAs have to consider infrastructure, environment, security, data management, backups, and workload-specific tuning. Further, maintaining and scaling Postgres involves meeting high availability requirements, managing data storage, updating schemas, optimizing query performance, and managing failover protection and caching. Lastly, extensions exist to support additional functionality in Postgres but they are hard to discover, evaluate, certify and deploy.

Put simply, Postgres is the solution.. if you can tame it and make it manageable.

Our Vision

This is where Tembo comes in. Tembo is the managed cloud to run Postgres and its entire ecosystem—extensions, applications, tools, and more—all within a single unified platform. It simplifies deploying Postgres with a virtualized runtime experience, enabling one-click migrations and access to the Postgres ecosystem. Developers can control the data model lifecycle, and deploy to multiple zones and clouds. Advanced options will include autoscaling, hybrid transactional and analytical processing (HTAP), and edge caching.

Additionally, Tembo invests in the Postgres extension ecosystem, aiming to standardize and simplify the use and creation of extensions. By unbundling and decoupling the database into services and abstraction layers, Tembo enables new simplicity and capability.

Tembo Cloud

It's Postgres the way it should be. With Tembo, you don't have to be a database expert to build an expert database. We are building a dev-first, fully-extensible, fully-managed, secure, and scalable Postgres service. Available on all clouds and bare metal providers, Tembo Cloud provides the largest curated library of easily-installed extensions, allowing our customers to expand their use cases of Postgres.

Organization home

Instance home

Tembo Stacks

It's “Postgres for Everything,” and we mean it. Stacks accelerate your development by enabling you to quickly create and deploy custom-built "flavors" of Postgres + extensions that are tailor made for key enterprise needs. No need to spin up new databases and endure the pain and associated sprawl—Stacks enable you to replace external, non-Postgres data services.

What Does This Mean?

No matter who you are or what you're trying to build, three things are true about Tembo:

  1. True Managed Open Source: You don't have to settle for a complex web of OSS data services or a restrictive, locked-in, expensive managed contract with one of the large cloud providers. Tembo is committed to making true open-source Postgres manageable and accessible.
  2. Developer Centric: You can have the flexibility and control you've dreamed of. Tembo is made by developers, for developers; we give you fast deployment, automatic migration, and a clear path to genuine value. We win when you win.
  3. Future Proof: Postgres is the world's most developer beloved database. It isn't going anywhere, and with the constantly growing ecosystem of extensions and applications, it's only getting better. With Tembo, you get all the potential of that ever-growing network, right at the click of a button.

Tembo: It's everything you need to build anything on Postgres.

· 3 min read
Ry Walker

pink nodes

Hello, world :) We are thrilled to announce the launch of our new startup, Tembo, and our mission to build a game-changing managed Postgres service.

Background

While wrapping up my involvement in my previous company, I learned about the rich Postgres extension ecosystem, and was drawn to an idea that these extensions represent a big opportunity to expand the use cases for Postgres—maybe even collapse the modern data stack into a Postgres-centric platform, and eliminate a lot of unnecessary and expensive data movement.The vast majority of developers and companies primarily use "Vanilla" Postgres. This is partly because the current user experience of discovering, installing, and gaining proficiency in Postgres extensions is subpar—especially compared to other ecosystems. This has been voiced by many community members. As such, most existing managed services offer only a sliver of the potential value that extensions could provide.

We've started a company

Tembo’s mission is to continuously improve the dev experience of Postgres, especially in the area of expanded use cases that rely on community extensions.Postgres is the most loved database by a long shot, and all Postgres users (including me) owe a huge debt of gratitude to all the companies and people who have invested into Postgres over the past 26 years. Our goal is to serve the Postgres community in an increasingly valuable manner.Our mission is to empower developers and businesses worldwide with the full power of Postgres and its extensions ecosystem, enabling users to use Postgres for a growing number of use cases.

We've raised some money

We have raised a $6.5M seed round led by Venrock, with participation from CincyTech and Wireframe Ventures, and other influential tech angel investors, and we've already got a team of 10 working on our SaaS product and related open-source projects. We've been building a product

We are building what we hope will be a world-class managed Postgres service:

  • Use any extension, even custom or private extensions
  • Provide UI for popular extensions, and a framework to include the community in those efforts
  • Stacks - packaging of extensions, tools and configurations, to make it easy to get started using Postgres for specific new use cases
  • An intelligent advisor that helps you to better use the features that you’re already using in Postgres

create cluster

org home

The Tembo team is incredibly excited to share our platform with you as we enter our closed beta period, and support your journey towards data-driven success.

We invite you to try Tembo Cloud, to access the future of database management.