querybara transfer

querybara transfer copies tables, collections or keys from a source database to a target in one streaming run. Rows move in batches, a transaction per batch, and several tables run at once. The two sides can be different engines: the CLI maps the column types per engine pair and reshapes data between tables and documents.

Usage

querybara transfer --help

Usage: querybara transfer [options] <source> <target>

copy tables, collections or keys from one database to another

Arguments:
  source                             profile name or id, or connection URI, to
                                     read from
  target                             profile name or id, or connection URI, to
                                     write to

Options:
  --table <name>                     table or collection to transfer
                                     (schema.table on PostgreSQL); repeatable
  --all                              every table of the source schema, or
                                     collection of the database
  --pattern <glob>                   Redis: keys to copy, e.g. "user:*";
                                     repeatable
  --database <name>                  source database (MongoDB: of the
                                     collections)
  --schema <name>                    PostgreSQL source schema (default public)
  --target-database <name>           target database
  --target-schema <name>             PostgreSQL target schema (default public)
  --mode <mode>                      create each table, drop and create it,
                                     empty it, or append (choices: "create",
                                     "drop-create", "truncate", "append",
                                     default: "create")
  --rename <from=to>                 target name of a table; repeatable
  --type <table.column=type>         target type of a column; repeatable
  --skip <table.column>              leave a column out; repeatable
  --shape <collection.field=shape>   MongoDB to SQL: a field as columns, json or
                                     a child table; repeatable
  --embed <parent:child:fk[:field]>  SQL to MongoDB: embed the child rows of
                                     each parent (by foreign key); repeatable
  --batch-size <n>                   rows, documents or keys per batch (default
                                     1000)
  --parallel <n>                     tables transferred at once (default 2)
  --on-error <action>                stop at the first failed row, or log it and
                                     go on (choices: "stop", "skip", default:
                                     "stop")
  --no-transaction                   no transaction per batch
  --disable-constraints              foreign key checks (PostgreSQL: and
                                     triggers, needs superuser) off during the
                                     load
  --no-defer-constraints             create keys, indexes and foreign keys
                                     before the data
  --no-reset-sequences               leave sequences and AUTO_INCREMENT counters
                                     as they are
  --sample <n>                       MongoDB: documents sampled for the columns
                                     (default 1000)
  --no-id-from-key                   SQL to MongoDB: do not make the primary key
                                     the _id
  --replace                          Redis: overwrite keys that exist on the
                                     target
  --no-ttl                           Redis: copy keys without their time to live
  --dry-run                          print the plan (column types, statements)
                                     and change nothing
  --json                             print the plan or the summary as JSON on
                                     stdout
  --error-log <file>                 write every failed row and its error to a
                                     file
  -y, --yes                          confirm dropping, emptying or overwriting,
                                     and production targets
  --tls <mode>                       TLS mode for this run: disable, require,
                                     verify-ca or verify-full
  --ssh <user@host[:port]>           reach URI targets through this SSH server;
                                     repeat for jump hosts, in order
  --ssh-key <path>                   SSH private key file (OpenSSH, PEM or PuTTY
                                     .ppk)
  --ssh-password-env <VAR>           take the SSH password from this variable
                                     (default QUERYBARA_SSH_PASSWORD, else a
                                     prompt)
  --ssh-agent                        log in with the keys of ssh-agent
                                     (SSH_AUTH_SOCK) or Pageant
  --proxy <url>                      reach URI targets (or their first SSH
                                     server) through socks5://host:port or
                                     http://host:port
  --ssh-accept-new                   trust and remember an SSH host key not seen
                                     before (a changed key is always refused)
  --known-hosts <path>               SSH known hosts file (default: the desktop
                                     app's)
  -h, --help                         show help for a command

Pairs: PostgreSQL, MySQL and MariaDB to any of them; those to MongoDB (typed documents, child
rows embedded with --embed); MongoDB to them (nested fields flattened to columns, arrays as
child tables or JSON, types from a sample); Redis to Redis (DUMP/RESTORE with TTLs, keys by
--pattern, standalone or Cluster). Rows stream in batches with a transaction per batch;
tables run --parallel at once on their own sessions, and primary keys, indexes and foreign
keys follow the data. The type of each column comes from a mapping per engine pair: see it
with --dry-run, change it with --type.

Exit codes: 0 transferred, 1 transferred but rows were skipped, 2 failed or refused, 130
interrupted. Safety: read-only targets refuse; drop-create, truncate, --replace, production
and "confirm writes" profiles need --yes or a confirmation.

Examples:
  querybara transfer pg-dev "mysql://app@localhost/shop" --table orders --table customers
  querybara transfer pg-dev my-dev --all --mode drop-create --yes
  querybara transfer my-dev mongo-dev --table orders --embed orders:items:items_ibfk_1:lines
  querybara transfer mongo-dev pg-dev --table events --shape events.tags=json --dry-run
  querybara transfer redis-a redis-b --pattern "session:*" --replace

The --tls and SSH options are described in Global options.

Engine pairs

Source	Target	What happens
PostgreSQL, MySQL or MariaDB	PostgreSQL, MySQL, MariaDB	Types mapped per engine pair; keys, indexes and foreign keys after the data
PostgreSQL, MySQL or MariaDB	MongoDB	Typed documents; child rows embedded through a foreign key with --embed
MongoDB	PostgreSQL, MySQL, MariaDB	Nested fields flattened to columns; arrays as child tables or JSON; types from a sample
Redis	Redis	DUMP/RESTORE with TTLs, keys chosen by --pattern, standalone or Cluster

Other pairs are refused.

Examples

Copy two tables from PostgreSQL to MySQL:

querybara transfer shop-pg "mysql://[email protected]/shop" --table orders --table customers

Copy every table of the source schema, dropping and recreating the targets:

querybara transfer shop-pg shop-mysql --all --mode drop-create --yes

Embed each order’s items in the order document:

querybara transfer shop-mysql shop-mongo --table orders --embed orders:order_items:order_items_ibfk_1:items

Check the plan for a MongoDB collection before copying it, with an array kept as JSON:

querybara transfer shop-mongo shop-pg --table events --shape events.tags=json --dry-run

Copy keys between Redis servers:

querybara transfer cache-a cache-b --pattern "cart:*" --replace

Plan and type mapping

--dry-run prints the plan, with the column types and statements, and changes nothing. Change a column’s target type with --type table.column=type, leave a column out with --skip, and rename a target table with --rename from=to.

--mode decides what happens to a target table: create (default), drop-create, truncate or append. Primary keys, indexes and foreign keys are created after the data unless you pass --no-defer-constraints. Sequences and AUTO_INCREMENT counters are reset after the load unless you pass --no-reset-sequences.

Safety

• Read-only targets refuse.
• drop-create, truncate, --replace, and production and confirm-writes profiles need a confirmation or --yes.
• --disable-constraints turns foreign key checks off during the load (on PostgreSQL also triggers, which needs a superuser).

Data loss

--mode drop-create and --mode truncate remove the target’s existing rows. On Redis, --replace overwrites keys that exist on the target.

Exit codes

Code	Meaning
0	Transferred
1	Transferred, but rows were skipped
2	Failed or refused; for --dry-run, the plan has problems
130	Interrupted

Related

• querybara import
• querybara data-compare
• querybara backup