Following some recent conversations on the subject of ephemeral instances, it looks there exist various misconceptions. Some people are trying to build ephemeral infrastructure but do not necessarily realize what is involved.
This post aims to clarify three major problems that you should address when dealing with ephemeral instances. These problems are: Infrastructure, Code and Data. Below I will clarify the meaning of each and some available tools – including those we are working on at Reliza.
I Infrastructure
An ephemeral instance relies on a set of infrastructure resources to support it. Usually those resources are maintained as infrastructure-as-code using tf files. When spinning an ephemeral, the goal is to have its whole infrastructure operational via single tofu apply instruction. Refer to our Open Source REEF project as a sample tool supporting such approach.
When using Kubernetes, sometimes an ephemeral infrastructure could be simplified to spin a separate namespace in the same cluster. But even in this case we need consistent tooling to make the process seamless.
Note here that the ephemeral infrastructure does not have to match production infrastructure exactly. In example, in production we may be using PostgreSQL database via managed service, while for an ephemeral instance we may use a Postgres container. This is fine as long as the difference is documented and does not interfere with the main purpose of such ephemeral.
II Code
Once our infrastructure is figured out, we need to be able to deploy specific versions of code artifacts.
The difficulty in modern distributed systems lies in the fact that there are many components involved. Therefore, when working with an ephemeral we need a clear understanding which parts of software should be on the same versions as production and what components should be different.
To some degree this may be solved via using GitOps, but such approach is not ideal at scale.
Our platform Reliza Hub is designed to address this problem. It maintains release records of every software component and their permutations. This allows the user to either match their production or staging environment or build a custom set of components with new changes, depending on the need.
III Data
The final piece of the puzzle is data. An ephemeral must use its own database to avoid cross-contamination of data.
Schema and seed data should be controlled within the migration system alongside code. I usually recommend Flyway for this task. If Flyway or a similar system is used, schema and seed data are synchronized naturally across all available instances. This solves the problem of database schema for ephemerals.
But there is more to the data problem. Sometimes, the users of ephemerals require some sample test data on top of the schema and seed data.
Producing such data as a difficult problem. Ideally, we would like to use real production data for this process. But production data is protected and it is usually not possible to transfer it to an ephemeral for confidentiality reasons. One approach to solve this problem is to maintain a custom set of sample data – that would be manually curated and similar to production data. However, this usually comes at a high cost and as a result rarely implemented.
Another approach is to obfuscate real production data to the point that it is not recognizable and use that for ephemerals. There is some tooling available for this – particularly RepliByte and Airbyte. Unfortunately, available tools also serve other purposes and require a lot of custom code to fit into the platform for spinning ephemerals. Therefore, the Data problem for ephemerals remains largely unsolved.
IV The Ephemeral problem
Having solved all three problems mentioned above you should have great working ephemeral system. But the big problem is they need to be solved all at the same time and in the “push-button” manner.
As mentioned above no tool available today is able to combine all these things at once. With Reliza Hub we are mostly at a stage where we can do Infrastructure, Code and Seed Data for most projects, but not sample data on its own. I.e., see my tutorial on how we can spin ephemerals here.
Therefore due to lack of universal tooling, most in-house solutions assume some compromises – usually requiring set of manual steps to replace missing tools.
However, it is important to note that while certain things could be replaced with manual steps, the basic structure of the application must be ephemeral-ready. Meaning, an application needs a well-defined infrastructure-as-code templates, it needs good governance of its release management process and it needs proper migration system for its database schema and seed data. Preparing the application to meet these criteria must be an absolute first step in the process.