Delphix Continuous Data Integration

“You never develop code without version control, why do you develop your database without it?”

- Alex Soto

Applications are the nexus of the modern enterprise. They simplify operations, speed execution, and drive competitive advantage. Accelerating the application lifecycle means accelerating the business. Organizations depend on rapid iteration as the foundation of agile development. This had led to a rich ecosystem of source code management tools that enable developers to work quickly within a private sandbox and push changes to a shared branch where continuous integration tools build and validate those changes on an ongoing basis.

This source code management agility has historically been stymied by data dependencies that are not easily moved, refreshed, or managed. This forces shared sandboxes, manual schema management, and stale data that degrades quality and slows development. Over the last decade, momentum has built around concepts such as Evolutionary Database Design and Database Continuous Integration, and an increasingly rich toolchain has emerged. While implementations vary, they share the same basic idea:

Structural (DDL) and data (DML) changes must be managed in concert with source code and applied in a rigorous and automated fashion.

These tools (flyway and liquibase being the most prominent) connect to arbitrary databases and manage the contents of the database, but make no attempt to manage the databases themselves. While they are capable of upgrading production data at any version, users typically operate on an empty database and populate it with synthetic data. While this is useful for unit tests that require repeatable results, it is a poor solution for functional, regression, and system test environments. Running those tests on synthetic, partial, or stale data can lead to bugs caught late in the development cycle, requiring costly process resets that could have been avoided if the developer had been using real data in their own sandbox. The result is longer application development cycles expensive process to manually push real data earlier into the lifecycle, and slower velocity for the business.

Continuous Integration with Delphix

Delphix is the engine that can accelerate these tools and provide a robust foundation for a new generation of continuous data integration processes. Database continuous integration tools provide the framework for DDL and DML change management, Delphix efficiently provides fresh real data to developers at the point they need it. Imagine a developer writing a DML translation based on the assumption about the format of some data. All of their manual, unit, and integration tests pass based on the synthetic data built on those same assumptions (or pulled from stale production data), but the application fails in UAT when confronted with an unknown format on the latest production data.

Bug Impact to Test Cycle Before Delphix

Bug Impact to Test Cycle After Delphix

Before Delphix, these bugs force the project team into an unpleasant situation: re-run the entire test cycle and lose weeks of development time, or run only a subset of tests after fixing the issue and jeopardize quality by risking bugs that might only be caught through a full test cycle. After Delphix, no one ever notices a bug that never escapes the developer environment.

Database continuous integration tools like flyway and liquibase are a perfect match for Delphix-driven provisioning and refresh. Each of these tools tracks ordered transformations using a table within the database to track what changes have been applied. This allows the software to bring any database up to the current developer version. There are no additional hooks required on the Delphix side, but there are ways Delphix can provide benefits beyond just fresh real data.

Developer Reset

These DDL and DML transformations are not foolproof. In some cases, it is necessary to write custom product code to run transformations that cannot be accomplished via SQL (such as updating BLOBs), or assemble a particularly tricky set of SQL statements. The developer should be able to test and validate these transformations on real data prior to integrating the changes. With Delphix, developers can tag their branch after all the known transformation have been run, and then rapidly iterate on validating new transformations with real data, resetting on failure without needing to refresh or get a new copy of real data.

Project Branches

Developers rarely work in isolation. Project or release branches allow developers to share code changes by pushing to a common source code repository. With Delphix-enabled continuous data integration, it’s possible for developers to share not just a source code repository, but a data repository as well. While continuous integration tools can apply changes within each developer copy, these transformations may time to run and consume additional storage depending on the size of the database and nature of the transformations. These changes can be automated within a project branch such that developers always get the latest transformed data whenever they refresh.

On refresh of a project branch, a script is invoked that transforms the production data using the latest DDL and DML changes pulled from the project source code repository. Combined with a refresh policy, this can keep the project branch up to do date with fresh data without each developer needing to run the same transformations. This process can also be run through hooks whenever a DDL or DML change is pushed to the source code repository.

Summary

Agile development requires agile data, and Delphix is the agile data engine of the future. You can sync to your production or source databases, instantly provision virtual databases where they are needed using a minuscule amount space, and provide each developer an independent copy of the data that can be refreshed and reset on demand. Combined with database continuous integration tools like flyway and liquibase, Delphix simplifies the development and testing of structural and data transformations. Providing efficient full data copies through Delphix that are always in sync with source code changes reduces bugs and accelerates application delivery while reducing IT and infrastructure overhead.

Agile Data Technology

The Requirements for Agile Data

Applications are the nexus of the modern enterprise. They simplify operations, speed execution, and drive competitive advantage. Accelerating the application lifecycle means accelerating the business. Increasingly, organizations turn to public and private clouds, SaaS offerings, and outsourcing to hasten development and reduce risk, only to find themselves held hostage by their data.

Applications are nothing without data. Enterprise applications have data anchored in infrastructure, tied down by production requirements, legacy architecture, and security regulations. But projects demand fresh data under the control of their developers and testers, requiring processes to work around these impediments. The suboptimal result leads to cost overruns, schedule delays, and poor quality.

Agile development requires agile data. Agile data empowers developers and testers to control their data on their schedule. It unburdens IT by efficiently providing data where it is needed independent of underlying infrastructure. And it accelerates application delivery by providing fresh and complete data whenever necessary. It grants its users super powers.

Many technologies can solve part of the agile data problem, but a partial solution still leaves you with suboptimal processes that impede your business. A complete agile data solution must embrace the following attributes.

Non Disruptive Synchronization

Production data is sensitive. The environment has been highly optimized and secured, and its continued operation is critical to the success of the business - introducing risk is unacceptable. An agile data solution must automatically synchronize with production data such that it can provide fresh and relevant data copies, but it cannot mandate changes to how the production environment is managed, nor can its operation jeopardize the performance or success of business critical activities.

Service Provisioning

Data is more than just a sequence of bits. Projects access data through relational databases, NoSQL databases, REST APIs, or other APIs. An agile data solution must move beyond copying the physical representation of the data by instantiating and configuring the systems to access that data. Leaving this process to the end users induces delays and amplifies risk.

Source Freedom

Data is pervasive. Efforts to mandate a single data representation, be it a particular relational or NoSQL system, rarely succeed and limit the ability of projects to choose the data representation most appropriate for their needs. As projects needs diversify the data landscape, the ability to manage all data through a single experience becomes essential. This unified agile experience necessitates a solution not tied to a single data source.

Platform Independence

The premier storage, virtualization, and compute platforms of today may be next year’s legacy architecture. Solutions limited to a single platform inhibit the ability of organizations to capitalize on advances in the industry, be it a high performance flash array or new private cloud software. Agility over time requires a solution that is not tied to the implementation of a particular hardware or software platform.

Efficient Bridges

Enterprise infrastructure spans geography, platform, and technology. Data anchored in the world's best datacenters is useless to the developer that needs it in the cloud, a project that needs it migrated to a different platform, or the tester that needs a local copy halfway around the world. Data mobility through efficient transfer mechanisms is required to get data where it needs to be.

Efficient Copies

Storage costs money, and time costs the business. Agile development requires a proliferation of data copies for each developer and tester, magnifying these effects. Working around the issue with partial data leads to costly errors that are caught late in the application lifecycle, if at all. An agile solution must be able to create, refresh, and rollback copies of production data in minutes while consuming a fraction of the space required for a full copy.

Workflow Automation

Each development environment has its own application lifecycle workflow. Data may need to be masked, projects may need multiple branches with different schemas, or developers may need to restart services as data is refreshed. Pushing responsibility to the end user is error prone and impedes application delivery. An agile solution must provide stable interfaces for automation and customization such that it can adapt to any development workflow.

Self Service Data

Developers and testers dictate the pace of their assigned tasks, and each task affects the data. Agile development mandates that developers have the ability to transform, refresh, and roll back their data without interference. This experience should shield the user from the implementation details of the environment to limit confusion and reduce opportunity for error.

Resource Management

Each data copy consumes resources through storage, memory, and compute. Once developers experience the power of agile data, they will want more copies, run workloads on them for which they were not designed, and forget to delete them when they are through. As these resources become scarce, the failure modes (such as poor performance) become more expensive to diagnose and repair. Combatting this data sprawl requires visibility into performance and capacity usage, accountability through auditing and reports, and proactive resource constraints.

Delphix is the agile data platform of the future. You can sync to your production or source databases, instantly provision virtual databases where they are needed using a minuscule amount space, and provide each developer an independent copy of the data that can be refreshed and rolled back on demand. This platform will become only more powerful over time as we add new data sources, provide richer workflows targeting specific applications and use cases, and streamline the self service model. An enterprise data strategy without Delphix is just a path to more data anchors, necessitating suboptimal processes that continue to slow application development and your business.

Behind the Music: The Delphix node.js CLI

As I indicated in a previous post, the new Delphix CLI is a node.js application that runs locally when the user logs into the system. User documentation can be found in the CLI User Guide, but I thought it would be interesting to explore how we ended up with the CLI we have today given the underlying web services upon which it is based.

Our web services are all defined by a set of schemas that I hope to one day describe in more detail. For those of you with a delphix server on hand, you can access the full schema at /api/json/delphix.json on any server. These schemas are loaded by the CLI to dynamically generate the content such that we don't need to update the CLI with every change made to the web services layer. Our APIs are generally RESTful with CRUD semantics, though they have a few oddities that are worth noting:

• We support singleton objects (i.e. NDMP Configuration) that export read/update on a global path.
• Our objects are polymorphic and leverage inheritance, so that we can have a set of operations (link, provision, delete, etc) that operate on a variety of types (Oracle, MSSQL, etc) without needing an entirely separate namespace (which would prevent consumers from generically iterating over objects).
• We support non-CRUD operations both per-object (start, stop, etc), and globally (link, provision, etc).
• We have persistent references to objects that are independent of the object name.
• Some operations execute asynchronously from the web service call and can take a while to run.

The CLI is very much inspired by Bryan's work on the Fishworks CLI (Fishworkers will undoubtedly take heart in the fact that the CLI docs still pay homage to dory and kiowa). We knew that a modal CLI based on filesystem-like navigation, property manipulation, and rich built-in help and tab completion would be vastly easier to use than a never-ending list of commands with esoteric --options-with-ridiculous-names. But we could also learn from the Fishworks CLI: because the underlying web service layer was not RESTful and well-formed, it meant that maintaining the CLI was quite expensive and semantics varied in subtle ways between contexts.

The first decision was to have the CLI namespace mimic that of the web services layer. If an API was rooted at /resources/json/delphix/service/smtp, it meant that the CLI location would be service smtp. This encourages both rational layout of APIs, and eliminates the need for extra translation. We allow for these paths to be typed in directly, but also support the cd command and UNIX-like paths for those familiar with those shells:

    delphix> database
    delphix database> cd /service/ndmp
    delphix service ndmp> cd ..
    delphix service>

For groups of objects, we have a list command (as well as a ls alias that shows everything at the current context) that can display objects, and a select command to select an individual node:

    delphix database> list
    NAME  PARENTCONTAINER  DESCRIPTION
    dory  -                -
    delphix database> select dory
    delphix database "dory"> cd /
    delphix> database "dory"
    delphix database "dory">

Now that we could move around the namespace, the next question was how to model operations: create, read, update, delete, and custom global and per-object operations. Each operation, regardless of HTTP implementation (PUT, POST, or DELETE) can take a single JSON object as input. Rather than specifying a complete object on the command line, we place the user into a context where they can interactively change properties and the elect to commit (or discard) the operation.

    delphix database> link
    delphix database link *> get container
        type: OracleDatabaseContainer
        name: (required)
        description: (unset)
        diagnoseNoLoggingFaults: true
        group: (required)
        masked: (unset)
        performanceMode: (unset)
    delphix database link *> set container.name=fluke
    delphix database link *> set dbUser=delphix

The APIs support nested objects as input, so the CLI uses dot-delimited properties (and tab completion) for specifying input. For objects that support multiple types (through inheritance), the set of available properties changes whenever the type parameter is changed.

For programmatic consumers, data can be output in JSON, and the 'trace' option can be turned on to see the HTTP calls made as part of an operation. Users can then experiment in the CLI, with a more natural interface, tab completion, integrated help, and then use the switch to the raw web services when more advanced manipulation is required.

There are many more topics in the world of the CLI and web services, from objects references in names to asynchronous jobs, to integration with the illumos PAM stack to hook into SSH authentication. But for my next post I hope to dive into the structure of the web services schemas, and describe how we use them on the backend, GUI, documentation, and CLI to automate significant parts of the engineering process.

Engineer Anti-Patterns

The other week I had a particularly disappointing discussion with a potential new hire. I typically describe our engineering organization at Delphix as a bottoms-up meritocracy where smart people seize opportunities to impact the company through through thoughtful execution and data-driven methodology (a.k.a. buzzword bingo gold). In this case, after hours of discussions, I couldn't shake this engineer's fixation with understanding how his title would affect his ability to have impact at Delphix. After deciding that it was not a good cultural fit, I spent some time thinking about what defines our engineering culture and what exactly it was that I felt was such a mismatch. Rather than writing some pseudo-recruiting material extolling the virtues of Delphix, I thought I'd take a cue from Bryan's presentation on corporate open source anti-patterns (video) and instead  look at some engineering cultural anti-patterns that I've encountered in the past. What follows is a random collection of cultural engineering pathologies that I've observed in the past and have worked to eschew at Delphix.

The Thinker

This engineer believes his responsibility is to think up new ideas, while others actually execute those ideas. While there are certainly execution-oriented engineers with an architect title out there that do great work, at Delphix we intentionally don't have such a title because it can send the wrong message: that "architecting" is somehow separate from executing, and permission to architect is something to be given as a reward. The hardest part of engineering comes through execution - plotting an achievable path through a maze of code and possible deliverables while maintaining a deep understanding of the customer problem and constraints of the system. It's important to have people who can think big, deeply understand customer needs, and build complex architectures, but without a tangible connection to the code and engineering execution those ideas are rarely actionable.

The Talker

Often coinciding with "The Thinker", this engineer would rather talk in perpetuity rather than sit down and do actual work. Talkers will give plenty of excuses about why they can't get something done, but the majority of the time those problems could be solved simply by standing up and doing things themselves. Even more annoying is their penchant for refusing to concede any argument, resulting in orders of magnitude more verbiage with each ensuing email, despite attempts to bring the discussion to a close. In the worst case the talker will provide tacit agreement publicly but fume privately for inordinate amounts of time. In many cases the sum total of time spent talking about the problem exceeds the time it would take to simply fix the underlying issue.

The Entitled

This engineer believes that titles are granted to individuals in order to expand her influence in the company; that being a Senior Staff Engineer enables her to do something that cannot be accomplished as a Staff Engineer. Titles should be a reflection of the impact already achieved through hard work, not a license granted by a benevolent management. When someone is promoted, the reasons should be obvious to the organization as a whole, not a stroke of luck or the result of clever political maneuvering. Leadership is something earned by gaining the respect of your peers through execution, and people who would use their title to make up for a lack of execution and respect of their peers can do an incredible amount of damage within an enabling culture.

The Owner

This engineer believes that the path to greater impact in the organization is through "owning" ideas and swaths of technology. While ownership of execution is key to any successful engineering organization (clear responsibility and accountability are a necessity), ownership of ideas is toxic. This can lead to passive-aggressive counter-productive acts by senior engineering, and an environment where junior engineers struggle to get their ideas heard. The owner rarely takes code review comments well, bullies colleagues that encroach on her territory, and generally holds parts of the product hostage to her tirades. Metastasized in middle management, this leads to ever growing fiefdoms where technical decisions are made for entirely wrong organizational reasons. Ideas and innovation come from everywhere, and while different parts of the organization are better suited to execution of large projects based on their area of expertise, no one should be forbidden from articulating their ideas due to arbitrary assignment.

The Recluse

Also known as "the specialist", this engineer defines his role in the most narrow fashion possible, creating an ivory tower limited by his skill set and attitude. Good engineers seize a hard problem and drive it to completion, even when that problem pushes them well beyond their comfort zone. The recluse, however, will use any excuse to make something someone else's problem. Even when the problem falls within his limited domain, he will solve only the smallest portion of the problem, preferring to file a bug to have someone else finish the overall work. When confronted on architectural issues, he will often agree to do it your way, but then does it his way anyway. Months later it can turn out he never understood what you had said in the first place or discussed it in the interim, and by then it's too late to undo the damage done.

All of us have the potential for these anti-patterns in us. It's only through regular introspection and frank discussions with colleagues that we can hope to have enough self awareness to avoid going down these paths. Most importantly, we all need to work to create a strong engineering culture where it is impossible for these pathologies to thrive. Once these pathologies become a fixture in a culture, they breed similar mentalities as the organization grows and can be impossible to eradicate at scale.

A node.js CLI?

Over the past several months, one of the new features I've been working on for the next release is the development of the new CLI for our appliance. While the CLI is the epitome of a checkbox item to most users, as a completely different client-side consumer of web APIs it can have a staggering maintenance cost. Our upcoming release introduces a number of new concepts that required that we gut our web services and built them  from the ground up - retrofitting the old CLI was simply not an option.

What we ended up building was local node.js CLI that takes programmatically defined web service API definitions and dynamically constructs a structured environment for manipulating state through those APIs. Users can log in with their Delphix credentials over SSH or the console and be presented with this CLI via custom PAM integration. Whenever I describe this to people, however, I get more than a few confused looks and questions:

• Isn't node.js for writing massively scalable cloud applications?
• Does anyone care about a CLI?
• Are you high?

Yes node.js is so hot right now.  Yes, a bunch of Joyeurs (the epicenter of node.js) are former Fishworks colleagues. But the path here, as with all engineering at Delphix, is ultimately data-driven based on real problems, and this was simply the best solution to those problems. I hope to have more blog posts about the architecture in the future, but as I was writing up a README for our own developers, I thought the content would make a reasonable first post. What follows is an engineering FAQ. From here I hope to describe our schemas and some of the basic structure of the CLI, stay tuned.

Why a local CLI?

The previous Delphix CLI was a client-side java application written in groovy. Running the CLI on the client both incurs a cost to users (they need to download an manage additional software) as well as making it a nightmare to manage different versions across different servers. Nearly every appliance shipped today has a SSH interface; doing something different just increases customer confusion. The purported benefit (there is no native Windows SSH client) has shown to be insignificant in practice, and there are other more scalable solutions to this problem (such as distributing a java SSH client).

Why Javascript?

We knew that we were going to need to be manipulating a lot of dynamic state, and the scope of the CLI would remain relatively small. A dynamic scripting language makes for a far more compelling development environment for rapid development, at the cost of needing a more robust unit test framework to catch what would otherwise be compile time errors in a strongly typed statically compiled language. We explicitly chose javascript because our new GUI will be built in javascript, and this both keeps the number of languages and environments used in production to a minimum, as well allowing these clients to share code where applicable.

Why node.js?

We knew v8 was the best-of-breed runtime when it comes to javascript, and we actually started with a custom v8 wrapper. As a single threaded environment, this was pretty straightforward. But once we started considering background tasks we knew we'd need to move to an asynchronous model. Between the cost of building infrastructure already provided by node (HTTP request handling, file manipulation, etc) and the desire to support async activity, node.js was the clear choice of runtime.

Why auto-generated?

Historically, the cost of maintaining the CLI at Delphix (and elsewhere) has been very high. CLI features lag behind the GUI, and developers face additional burden to port their APIs to multiple clients. We wanted to build a CLI that would be almost completely generated from shared schema. When developers change the schema in one place, we auto-generate both backend infrastructure (java objects and constants), GUI data model bindings, and the complete CLI experience.

Why a modal hierarchy?

The look and feel of the Delphix CLI is in many ways inspired by the Fishworks CLI. As engineers and users of many (bad) CLIs, our experience has led to the belief that a CLI with integrated help, tab completion, and a filesystem-like hierarchy promotes exploration and is more natural than a CLI with dozens of commands each with dozens of options. It also makes for a better representation of the actual web service APIs (and hence easier auto-generation), with user operations (list, select, update) that mirror the underlying CRUD API operations.

Data Replication: Approaching the Problem

With our next Delphix release just around the corner, I wanted to spend some time discussion the engineering process behind one of the major new features: data replication between servers. The current Delphix version already has a replication solution, so how does this constitute a "new feature"? The reason is that it's an entirely new system, the result of an endeavor to create a more reliable, maintainable, and extensible system. How we got here makes for an interesting tale of business analysis, architecture, and implementation.

Where did we come from?

Before we begin looking at the current implementation, we need to understand why we started with a blank sheet of paper when we already had a shipping solution. The short answer is that what we had was unusable: it was unreliable, undebuggable, and unmaintainable. And when you're in charge of data consistency for disaster recovery, "unreliable" is not an acceptable state. While I had not written any of the replication infrastructure at Fishworks (my colleagues Adam Leventhal and Dave Pacheco deserve the credit for that), I had spent a lot of time in discussions with them, as well as thinking about how to build a distributed data architecture at Fishworks. So it seemed natural for me to take on this project at Delphix. As I started to unwind our current state, I found a series of decisions that, in hindsight, led to the untenable state we were in today.

• Analysis of the business problem - At the core of the current replication system was the notion that its purpose was for disaster recovery. This is indeed a major use case of replication, but it's not the only one (geographical distribution of data being another strong contender). While picking one major problem to tackle first is a reasonable approach to constrain scope, by not correctly identifying future opportunities we ended up with a solution that could only be used for active/passive disaster recovery.
• Data protocol choice - There is another problem that is very similar to replication: offline backup/restore. Clearly, we want to leverage the same data format and serialization process, but do we want to use the same protocol? NDMP is the industry standard for backups, but it's tailored to a very specific use case (files and filesystems). By choosing to use NDMP for replication, we sacrificed features (resumable operations, multiple streams) and usability (poor error semantics) and maintainability (unnecessarily complicated operation).
• Outsourcing of work - At the time this architecture was created, it was decided that NDMP was not part of the company's core competency, and we should contract with a third party to provide the NDMP solution. I'm a firm believer that engineering work should never be outsourced unless it's known ahead of time that the result will be thrown away. Otherwise, you're inevitably saddled with a part of your product that you have limited ability to change, debug, and support. In our case, this was compounded by the fact that the deliverable was binary objects - we didn't even have source available.
• Architectural design - By having a separate NDMP daemon we were forced to have an arcane communication mechanism (local HTTP) that lost information with each transition, resulting in a non-trivial amount of application logic resting in a binary we didn't control. This made it difficult to adapt to core changes in the underlying abstractions.
• Algorithmic design - There was a very early decision made that replication would be done on a per-group basis (Delphix arranges databases into logical groups). This was divorced from the reality of the underlying ZFS data dependencies, resulting a numerous oddities such as being unable to replicate non self-contained groups or cyclic dependencies between groups. This abstraction was deeply baked into the architecture such that it was impossible to fix in the original architecture.
• Implementation - The implementation itself was built to be "isolated" of any other code in the system. When one is replicating the core representation of system metadata, this results in an unmaintainable and brittle mess. We had a completely separate copy of our object model that had to be maintained and updated along with the core model, and changes elsewhere in the system (such as deleting objects while replication was ongoing) could lead to obscure errors. The most egregious problems led to unrecoverable state - the target and source could get out of sync such that the only resolution was a new full replication from scratch.
• Test infrastructure - There was no unit test infrastructure, no automated functional test infrastructure, and no way to test the majority of functionality without manually setting up multi-machine replication or working with a remote DMA. As a result only the most basic functionality worked, and even then it was unreliable most of the time.

Ideals for a new system

Given this list of limitations, I (later joined by Matt) sat down with a fresh sheet of paper. The following were some of the core ideals we set forth as we built this new system:

• Separation of mechanism from protocol - Whatever choices we make in terms of protocol and replication topologies, we want the core serialization infrastructure to be entirely divorced from the protocol used to transfer the data.
• Support for arbitrary topologies - We should be able to replicate from a host to any number of other hosts and vice versa, as well as provision from replicated objects.
• Robust test infrastructure - We should be able to run protocol-level tests, simulate failures, and perform full replication within a single-system unit test framework.
• Integrated with core object model - There should be one place where object definitions are maintained, such that the replication system can't get out of sync with the primary source code.
• Resilient to failure - No matter what, the system must be maintain consistent state in the face of failure. This includes both catastrophic system failure, as well as ongoing changes to the system (i.e. objects being created and deleted). At any point, we must be able to resume replication from a previously known good state without user intervention.
• Clear error messages - Failures, when they do occur, must present a clear indication of the nature of the problem and what actions must be taken by the user, if any, to fix the underlying problem.

At the same time, we were forced to limit the scope of the project so we could deliver something in an appropriate timeframe. We stuck with NDMP as a protocol despite its inherent problems, as we needed to fix our backup/restore implementation as well. And we kept the active/passive deployment model so that we did not require any significant changes to the GUI.

Next, I'll discuss the first major piece of work: building a better NDMP implementation.