A few months ago I got interviewed by a substack on open source development about Dolt. One of the questions I was glad to answer was what motivates me to contribute to Dolt. Well there’s the money, obviously. But more than that, it’s being part of a culture that ships.
At DoltHub, we ship major new features every month, and a stream of smaller improvements and bugfixes on a continual basis. We ship so often, in fact, that it can be hard to keep track of what we’ve accomplished and what we’re going to do next. This blog post is an attempt to pause and take stock of how far the product has come in the last couple years, and reflect on where we’re going next.
Product planning at DoltHub#
When we interview candidates, there’s always a period when we flip the process around and let them ask us questions instead. One of the most frequent questions we get from experienced candidates is: what sort of project planning process do you use? Because we’ve been in the industry for decades, we recognize this question for what it is: a desperate plea to not have to use Jira or play planning poker ever again.
I tell them it’s pretty simple. We have a big spreadsheet that we use to keep track of planned feature work, and every month or so we update it to reflect progress and priorities. It looks like this:
We add new features there when we decide to work on them, and move rows up and down to indicate their relative priorities. For time-sensitive work, we’ll sometimes add a deadline to a row. But that’s basically it, very little process. The main point of the spreadsheet is to enable us to have discussions about staffing tradeoffs and customer expectations.
- If we deliver stored procedures in October, what else has to slip?
- If Andy works on rewriting the storage layer, who will take over keyless table storage?
We also end up doing a lot of unplanned work, typically requested by a paying customer. They come to us with a query that’s too slow or doesn’t work correctly, and we give their issue top priority until we have a workaround. Sometimes those issues will get logged on GitHub, but often they never leave our customer chat rooms on Discord. And that’s fine. The point of the roadmap isn’t to perfectly capture all the work we do (release notes capture that pretty well), it’s to help us plan.
Victory lap#
At DoltHub, we ship a lot. We announce new features on this blog as they come out, but I want to take a moment to put all these accomplishments in one place, so that you (and we) can appreciate just how many there are. Warning: this is a long list!
| Feature | Release date |
|---|---|
| Change column type | Feb 2021 |
| Primary key changes | Jan 2021 |
| Indexes for keyless tables | Aug 2021 |
| parser allows reserved words as column names without backticks | May 2020 |
| explain statements show index usage | Aug 2020 |
| Outer scope accessible in subqueries | Aug 2020 |
| DESCRIBE TABLE, etc support for indexes and foreign keys | Sep 2020 |
| sqllogictest 99% | Mar 2021 |
| all information_schema tables present | Nov 2020 |
| Schema alteration on keyless tables | Aug 2021 |
| Column defaults | Sept 2020 |
| Triggers | Oct 2020 |
| Dolt CLI functions for SQL | Feb 2021 |
| Auto increment | Nov 2020 |
| Prepared statements | Nov 2020 |
| SIGNAL statement | Mar 2021 |
| Stored procedures | Mar 2021 |
| Keyless tables | Dec 2020 |
| Common table expressions (WITH) | Mar 2021 |
| Tuples for IN expression (multi-column IN) | Aug 2021 |
| Window functions (OVER) | Feb 2021 |
| dolt_commit_ancestors system tables | Mar 2021 |
| dolt_push() and dolt_pull() functions | Sep 2021 |
| Concurrency and transactions | May 2021 |
| INSERT…ON DUPLICATE | Mar 2020 |
| JSON type support | Apr 2020 |
| CHECK constraints | Apr 2021 |
| Foreign Keys | Jul 2020 |
| TRUNCATE table | June 2020 |
| Metaflow support | Apr 2021 |
| CREATE TABLE SELECT | Aug 2020 |
| Hash IN clause evaluation | Jan 2021 |
| N-table joins | Dec 2020 |
| Secondary indexes | May 2020 |
| Use more than one core | Feb 2020 |
| Push where clause down in join execution | Oct 2020 |
| Push projections to Dolt tables (return only a subset of columns) | Mar 2020 |
| Read from indexes, rather than full tables, when possible | Mar 2020 |
| MySQL Workbench support | Aug 2021 |
| Google Sheets support | June 2021 |
| Kedro Support | June 2021 |
| Great Expectations support | June 2021 |
| R Support | Aug 2021 |
| DataGrip support | May 2020 |
| Django support | Aug 2021 |
| Replication | Sept 2021 |
| Backup | Sept 2021 |
| Tags | Sept 2020 |
| Schema merge | May 2020 |
| Shallow pull, clone, fetch | Feb 2020 |
| filter-branch | Nov 2020 |
| Type conversion tests | Apr 2020 |
| dolt system tables | Jan 2020 |
| Detached HEAD SQL mode | Mar 2021 |
| Constraint violations | July 2021 |
| Check constraint violations command | Mar 2021 |
Support for main default branch | Sep 2021 |
| LOAD_FILE() support | Aug 2021 |
| Generational garbage collection | Aug 2021 |
| Ecto and Elixir support | July 2021 |
| Performance benchmarking | Oct 2020 |
| DoltHub forks | Sept 2020 |
| Query diff | June 2020 |
| Serving multiple databases in a single server | May 2020 |
| AS OF support | Mar 2020 |
| Saved queries | Feb 2020 |
| 2-table indexed joins | Feb 2020 |
| LICENSE and README files | Feb 2020 |
| Views | Feb 2020 |
| SQL queries on DoltHub | Jan 2020 |
| dolt blame | Oct 2019 |
And there’s a lot of stuff not even on this list, either because it got done without any fanfare or because it predates when we adopted even this limited planning process. DoltHub is a company that ships, a lot.
Today’s roadmap#
The product is a lot more mature today than a few years ago, as one would hope. In the earlier days there were so many missing features that prioritization was actually pretty easy: unless somebody was asking for a feature, we would be adding a lot of value no matter where we turned our attention, so strict prioritization didn’t matter too much. I joked about this situation in an earlier blog post, but having such a huge surface area to cover was actually really fun and made planning pretty easy.
Today things are a little harder. Besides being an environment that is less target-rich than before, we have a growing number of paying customers and their use cases to support, and a larger pool who would adopt the product if it had some capabilities it doesn’t yet. So it’s more important now to think about what we’re going to support next to make our existing customers happy and lure new ones.
This is always a work in progress, but here’s our current top priorities for Dolt:
| Feature | ETA |
|---|---|
| Hosted Dolt | Jan 2022 |
| Join for update | Oct 2021 |
| Backup and replication | Nov 2021 |
| Commit graph performance | Nov 2021 |
| Collation and charset support | Nov 2021 |
| Persistent SQL configuration | Dec 2021 |
| Multiple DBs in one repo | Dec 2021 |
| Tx isolation levels | Dec 2021 |
| 99.9% SQL correctness | Q1 2022 |
Better dolt_diff table experience | Q1 2022 |
| Hash join strategy | Q1 2022 |
| Storage performance | Q1 2022 |
| SQL GUI support tests | Q1 2022 |
| Lock / unlock tables | Q1 2022 |
| Users / grants | Q2 2022 |
JSON_TABLE() | Q2 2022 |
| Pipeline query processing | Q2 2022 |
| Table / index statistics | Q2 2022 |
| Universal SQL path for CLI | Q2 2022 |
| Row-level locking (select for update) | Q2 2022 |
| Virtual columns and json indexing | Q2 2022 |
| Embedded dolt | Q3 2022 |
| Signed commits | Q3 2022 |
This list is mostly ordered by planned release date, which gets less certain as we get farther out. Our top priority, hosted Dolt, is a relatively large effort and a major launch, scheduled for year end. Most of the other items on the list are a lot smaller, but there are exceptions: storage performance is code for a near-total rewrite of the storage layer to make it performant for the SQL server, which is a monumental effort (good luck, Andy).
We expect to rearrange this list as time goes on, and for new items to emerge and jump the line. Paying customers (or prospective paying customers) get write access to this roadmap, so if things go well this list will be obsolete in no time flat.
Conclusion#
DoltHub ships, a lot. We’re proud of the product features we’ve shipped so far, and eager to put more under our belt. If that sounds like an environment you’d like to be a part of, we’re hiring!
Like the article? Interested in Dolt? Think we should be working on other things instead? Come join us on Discord to say hi and let us know what you think.