SFTP and feature flags for Kipchak

Two drivers join the Kipchak ecosystem, and both are free. The SFTP driver exchanges files with systems that still trade in batch files, such as banks, payment processors, suppliers and logistics partners. The Feature Flags driver evaluates feature flags from Unleash, Flagsmith or GrowthBook, so that a feature can be released to some customers before others, or withdrawn, without deploying the API again.

Both were listed on the public roadmap. Each was tested in three stages: unit tests, then integration tests against the real server running in Docker, then a complete Kipchak API running under FrankenPHP worker mode while that server was stopped, paused and restarted.

The SFTP driver

Many organisations exchange data with their partners as files rather than through APIs: a bank delivers statements each night, a supplier collects purchase orders from a directory, a payroll provider expects a file in a fixed format by a fixed time. These exchanges usually run over SFTP. The SFTP driver lets a Kipchak API take part in them:

$sftp = Sftp::get('bank');

foreach ($sftp->files('inbound', 'statement-*.csv') as $file) {
    $statement = $sftp->read($file->path);
    // ... process the statement ...
    $sftp->move($file->path, 'archive/' . $file->name);
}

$sftp->write('outbound/payments-' . date('Ymd') . '.csv', $csv);

Each partner is a named connection in configuration, with its own host, credentials and directory.

The server is verified before credentials are sent

An SFTP connection is only as trustworthy as the server at the other end. The driver therefore requires the SHA256 fingerprint of the server’s host key, in the form that ssh-keyscan and ssh-keygen print. The fingerprint is checked before any password or key is sent. If the server presents a key that is not configured, the connection is refused and the exception includes the fingerprint that was presented, so that it can be compared with the one the partner publishes.

Private keys can be read from a file or supplied as the key itself, for example from a Kubernetes secret through an environment variable, so a key does not have to be written to disk.

A partner never collects a partial file

A program that collects files from a directory may read a file while it is still being written. The driver therefore writes every file under a temporary name in the same directory, checks that the size on the server matches what was sent, and only then renames it to its final name. On OpenSSH and other servers that support atomic renaming, a file that already exists is replaced in a single step. A write that fails removes its temporary file.

Downloads follow the same rule in the other direction: the local file is written under a temporary name and renamed when the download is complete. Uploads and downloads stream between disk and the server, so a large file does not have to fit in memory.

Paths are confined to a directory

A connection can be confined to a root directory. Every path is then resolved inside it, and a path that leads outside it, such as one containing ../, is rejected before anything is sent. A file name taken from a request cannot therefore reach the rest of the partner’s server.

Connections are kept between requests

Opening an SFTP connection involves a key exchange and authentication, which takes far longer than the file operation that follows. Under FrankenPHP’s worker mode, each worker opens a connection on first use and keeps it for later requests. SFTP servers close idle connections and are restarted for maintenance, so when an operation finds that its connection has closed, the driver opens a new one and repeats the operation once. Every operation is safe to repeat, because a repeated write starts again with a new temporary file.

The driver is built on phpseclib, which is written entirely in PHP and needs neither the ssh2 extension nor an ssh program. It therefore runs on the hardened runtime image, which contains no shell and no system utilities.

The SFTP driver is free and MIT-licensed:

composer require kipchak/driver-sftp

The configuration reference is in the SFTP driver documentation.

The Feature Flags driver

A feature flag separates releasing a feature from deploying the code that contains it. The code is deployed with the feature switched off, and the feature is then switched on for a group of customers, a percentage of users or everyone, from the flag service’s console. If it causes a problem, it is switched off again in the same place.

The driver supports three flag services: Unleash, Flagsmith and GrowthBook. Each connection in configuration names its service, and application code is the same whichever service a connection uses:

$flags = FeatureFlags::get();
$context = new Context(userId: $customer->id, attributes: ['plan' => $customer->plan]);

if ($flags->isEnabled('new-checkout', $context)) {
    // ...
}

$limit = $flags->value('export-limit', default: 1000, context: $context);
$variant = $flags->variant('pricing-page', $context);

The context describes who the flag is being evaluated for. The driver passes it to each service in the form that service expects: as Unleash context fields, as a Flagsmith identity and its traits, or as GrowthBook attributes.

Flags are evaluated within the API

Each service’s own PHP SDK is used to evaluate flags, so targeting rules, segments, percentage rollouts and experiments behave exactly as they are defined in the service’s console. The definitions are fetched at an interval, 30 seconds by default, and every check is then evaluated within the PHP process. A check therefore does not wait for a network request.

Under FrankenPHP’s worker mode, each worker keeps the definitions in memory between requests. The SDKs were written for PHP processes that last one request, and the GrowthBook SDK holds the current user’s attributes on a shared object. The driver evaluates each request on a separate copy of that object, so that one customer’s attributes cannot affect the answer given to the next. For APIs that do not run in worker mode, the definitions can be kept in Kipchak’s Filecache or Valkey drivers and shared between processes and machines.

The API continues to answer when the flag service is down

A flag check is made during the request it affects, so a flag service that is unavailable must not make the API unavailable. The driver behaves as follows:

After a successful fetch, the most recent definitions continue to be used for as long as the service is unavailable.
Before any fetch has succeeded, every check returns the default value given in the code, and the failure is logged at most once a minute.
A failed fetch is retried after 5 seconds, then after progressively longer intervals up to one minute. Between attempts, checks do not wait on the network.

We tested this with all three services paused. A paused service accepts connections but does not answer them, so each attempt waits for the full timeout. The API continued to give correct answers from the definitions it held, and resumed fetching within seconds of the services returning. A connection can instead be set to raise an exception when the service cannot be reached, for flags where no default is acceptable.

A file of flag definitions can also be given to each connection. It is used until the first fetch succeeds, and without a service address it is the only source, which suits automated tests and deployments without internet access.

The Feature Flags driver is free and MIT-licensed. Install it with the SDK of the service you use:

composer require kipchak/driver-feature-flags
composer require unleash/client     # or flagsmith/flagsmith-php-client, or growthbook/growthbook

The configuration reference, including how each service’s values and variants are returned, is in the Feature Flags driver documentation.

In the starter project

The Kipchak starter project includes an example of each driver. GET /v1/sftp lists the files on an SFTP server, and GET /v1/flags evaluates a set of sample flags from a file included in the project, so it works without a flag service.

A middleware that uses the Feature Flags driver to release features per API key is on the roadmap.

Both drivers are available now.