In the Inria’s Discovery initiative, we get in touch with CockroachLabs guys with an idea: make Keystone supports CockorachDB. So we give it a try and you can find a very first result on our GitHub. The GitHub project consists of a Vagrant file that spawns a VM with a CockroachDB database and then installs Keystone with Devstack using CockroachDB as backend.
CockroachDB claims to support the PostgreSQL protocol. It also provides support for SQLAlchemy that mostly inherits from the PostgreSQL one. So, making Keystone working with CockroachDB should be easy peasy right? Almost! The rest of this post starts by explaining why we choose Keystone for our POC. It then describes what we have done to make Keystone works over CockorachDB.
Focus on Keystone first, conquer all services then
OpenStack is a huge project, made of dozens of services. All services
are developed independently, but built with common principals. In
particular, a service stores its entire state in a database relying on
a library called
oslo.db that abstracts databases binding. Under the
oslo.db wraps SQLAclhemy. Thus, OpenStack supports MariaDB and
From all OpenStack services, we decided to focus first on Keystone.
Keystone is one of the core services of OpenStack. It manages all the
authentication part of the OpenStack process. We focus on Keystone
with the following idea: if we can make Keystone work with
CockroachDB, then we can hopefully make all services work with
CockroachDB thanks to
We choose Keystone because its DB requests are quite simple: Keystone splits the authentication management in 10 concerns (credentials, token, domain, …), whose code and DB requests are relatively easy to follow. Keystone presents another advantage when you want to test OpenStack at scale. To scale, current deployments put instances of OpenStack in different regions around the globe. Across these regions, operators use Galera to synchronise Keystones’ database. Thus, it makes authentication the same, wherever a client authenticates herself. However, Galera presents major limitations in case of high latency such as in Edge computing. So focusing on Keystone will let us test how the combo Keystone/CockroachDB outshines these limitations.
sqlalchemy-migrate doesn’t work with CockroachDB
Keystone uses a database migration tool named sqlalchemy-migrate. This tool is called during the deployment of Keystone to migrate the database from its initial versions to its actual version. Unfortunately, migration failed with the following (partial) stacktrace:
DEBUG migrate.versioning.repository [-] Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'keystone'), ('version_table', 'migrate_version'), ('required_dbs', ''), ('use_timestamp_numbering', 'False')]))]) __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/repository.py:83 INFO migrate.versioning.api [-] 66 -> 67... CRITICAL keystone [-] KeyError: 'cockroachdb' ... TRACE keystone visitorcallable = get_engine_visitor(engine, visitor_name) TRACE keystone File "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", line 47, in get_engine_visitor TRACE keystone return get_dialect_visitor(engine.dialect, name) TRACE keystone File "/usr/local/lib/python2.7/dist-packages/migrate/changeset/databases/visitor.py", line 62, in get_dialect_visitor TRACE keystone migrate_dialect_cls = DIALECTS[sa_dialect_name] TRACE keystone KeyError: 'cockroachdb'
Actually, sqlalchemy-migrate contains dedicated SQL backend codes and there is no such code for CockroachDB. So, we developed one. The code circumvents some CockroachDB limitations. For instance, CockroachDB does not support the add of Primary Key constraint. So, we implement this migration by, first making a new temporary table with the new primary keys. Then, copying elements of the original table into the temporary one. And finally, dropping the original table and renaming the temporary one. You can find that specific snippet at lines 238-257 of our fork of sqlalchemy-migrate.
Honestly, the support is only partial and not well-tested. A
-epy27 shows that the code successfully passes 90% of the unit tests,
but still fails on cleaning some index. However, our patch lets us
deploy Keystone over CockroachDB with Devstack, which is good enough
for our POC.
oslo.db contains backend specific code for error handling
oslo.db library contains one file with backend-specific codes
oslo_db/sqlalchemy/exc_filters.py). This file aims at abstracting
database exceptions that differ but target the same class of error
(because each backend produces a specific exception with a specific
error message). Intercepted exceptions are wrapped into a common OS
exception to abstract backend errors in the rest of
CockroachDB doesn’t produce same errors as PostgreSQL, so we have to
update this class. Note that our POC is not exhaustive since we only
added error messages we saw during Tempest/Rally tests.
You can look at the differences between OpenStack/oslo.db/stable/pike and our fork on GitHub. We only add two lines!
CockroachDB manages isolation without lock
CockroachDB lets you write transactions that respect ACID properties.
Regarding the “I” (Isolation), CockroachDB offers snapshot and
serializable isolation but doesn’t rely on locks to do that. So every
time there is concurrent editing transactions that end in a conflict,
then you have to retry the transactions. Fortunately,
already offers a decorator to do that automatically. But, based on
tests we run with Tempest/Rally, we figured out that some Keystone’s
SQL requests needed the decorator.
You can look at the differences between OpenStack/keystone/stable/pike and our fork on GitHub.
You can drop yourself on the VM as a stack user and run Tempest/Rally tests (see README). Note that modifications we made are minimal. This is promising for the adoption of CockroachDB by other core services. Nonetheless:
We have to look through the performance prism: Keystone over CockroachDB vs. Keystone over Galera. This part may involve more modifications of
oslo.db. One thing we have in mind is the management of retrying transactions since CockroachDB’s documentation suggests an approach that doesn’t match with actual
We recently start a similar work for Nova, the compute service of OpenStack. Nova produces SQL query that are far more complex than Keystone. In particular, Nova produces correlated subqueries, i.e., the subqueries are ran against each row returned by the main query. Unfortunately, correlated subqueries are not supported by CockroachDB. So keep in touch, we will see in a future post how we cope with these queries.