Tableflow Setup | WarpStream ⌘Ctrlk WarpStream.comSlackDiscordContact UsCreate AccountMore GitBook Assistant Good evening I'm here to help you with the docs. What is this page about?What should I read next?Can you give an example? ⌘Ctrli AI Based on your context Send Overview Introduction Architecture Change Log Getting Started Install the WarpStream Agent / CLI Run the Agents Locally "Hello World" for Apache Kafka Agent Setup Deploy the Agents Object Storage Configuration Set up Monitoring Infrastructure as Code Kafka Configure Clients Manage Security Manage Connectors Deploy (advanced) Reference Benchmark Orbit (Cluster Linking) Tableflow Tableflow Setup Monitoring Tableflow Iceberg REST Catalog Integrate With Query Engines and External Catalogs API Reference Schema Registry Setup Enforce Schemas Schema Linking Manage Security Reference Billing Audit Logs Events MCP Server Accessing the Confluent Support Portal Security and Privacy Considerations Manage Console Access Secrets Overview API Reference CLI Reference Integrations Dedicated Control Plane Cells and Release Windows Control Plane Private Connectivity Deprecation Policy Powered by GitBook On this page For the complete documentation index, see llms.txt. This page is also available as Markdown. Ask On this page Tableflow Tableflow Setup This page describes how to setup WarpStream Tableflow. Introduction Tableflow automates the tedious process of transforming a topic in an Apache Kafka-compatible data streaming system into an Apache Iceberg table. Instead of writing custom code and manually configuring a data pipeline for each table you want to build, Tableflow allows you to declaratively specify which topics to build tables from and what schema and data format to expect. When schemas inevitably need to change, you can update the schema in Tableflow's editor and WarpStream will handle the schema migration automatically. Compaction and table maintenance is included out-of-the-box with no tuning required. Tableflow continuously compacts the table in the background with intelligent heuristics to ensure readers get the best performance. Tableflow is available as Bring-Your-Own-Cloud (BYOC) where the compute and storage live inside your cloud account inside your VPC. The raw data for your table is only ever stored inside your object storage bucket and never leaves your VPC during the table ingestion and maintenance process. Tableflow maintains a metadata store inside WarpStream Cloud as the Iceberg metadata layer that is periodically synced into your object storage bucket. Getting Started To get started with Tableflow, you first need to create a Tableflow cluster from the WarpStream Console, or using one of infrastructure-as-code deployment options. The WarpStream Agents that join this cluster will only perform Tableflow operations and do not expose the Apache Kafka protocol. Please refer to our other documentation for how to install and configure the WarpStream Agents in your environment as the process does not differ for Tableflow. As part of deploying the Agents, you'll also need to setup and configure an object storage bucket and/or provide the Agents with access to one of your existing buckets. See our object storage configuration documentation for more details on that. Once the Agents are running, you can open the Configuration table and start defining your source clusters, topics, tables, and schemas. Managed Tables Tableflow tables are fully managed by WarpStream, or what we call "managed tables". You cannot use another system for performing writes, compactions, or other table maintenance operations. This is in contrast to a connector-based approach where you would be forced to combine multiple distinct systems or operations together to implement all of these functions. Configuration Tableflow is configured with and is fully controllable from a single YAML file which can be edited through the WarpStream console or the Pipelines API. Overview Currently, there are two methods for defining schemas: an inline mode which doesn't require using an external Schema Registry. This supports all schema types (JSON, Avro, Protobuf). The full schema must be fully contained in the YAML config: you cannot import external schemas with this mode. a schema_registry mode which requires using an external Schema Registry. Protobuf and JSON schemas are supported from agent version v813+, Avro schemas are supported from agent version v820+. When using the Schema Registry, a schema may reference and import other schemas (as long as they are also registered in the Schema Registry), and Tableflow will resolve the final schema. An example YAML for the inline mode is the following: An example YAML for the schema_registry mode is the following: The YAML specifies The source clusters Tableflow should connect to. The schema registries Tableflow should connect to (for the schema_registry mode). For each cluster, the topic that Tableflow should create Iceberg tables from. For each topic, the schema to deserialize the Kafka records with, either as an inline schema, or as referenced by a subject of a schema registry. The destination bucket to store the table. Configure Source Clusters Source clusters are the Apache Kafka-compatible systems like WarpStream that store the topics you'd like to convert to tables. You define clusters by giving them a name, a list of brokers, and credentials if they are needed. You define source clusters at the root of the configuration YAML. You can define multiple source clusters so a single Tableflow cluster can centralize data from multiple clusters into one unified place. Configure Schema Registries Schema registries are the Confluent-compatible schema registries that store the schemas you'd like to use to deserialize the data from your topics. In schema_mode: schema_registry, Protobuf and JSON tables require agent v813+ and Avro tables require agent v820+. You define schema registries by giving them a name, a URL and credentials if they are needed. You define schema registries at the root of the configuration YAML. All credential fields reference environment variable names. The fields you may define for the credentials are the following: Field Description username_env Env var holding the basic-auth username. Must be set together with password_env. password_env Env var holding the basic-auth password. Must be set together with username_env. use_tls Enables TLS / mTLS for the registry connection. mtls_client_cert_env Env var holding the path to the client certificate (PEM). Must be set together with mtls_client_private_key_env. mtls_client_private_key_env Env var holding the path to the client private key (PEM). Must be set together with mtls_client_cert_env. mtls_client_private_key_password_env Env var holding the private key password. Only allowed if mtls_client_private_key_env is also set. server_ca_cert_env Env var holding the path to the server CA certificate (PEM). Please note that for all those credentials, contrary to the Configure Connection and Credentials section below, the agents do not automatically add a TABLEFLOW_ prefix to the values of the fields before using the environment variables. So the environment variables in the Agent should be exactly the same as those stated in the configuration.