Kipchak has three new packages. The NATS driver is free and open source under the MIT licence. The PII Redaction and Audit Log middlewares are part of Kipchak Enterprise. All three were listed on the public roadmap.
The NATS driver
NATS is a single, small server that provides messaging between services, work queues, durable streams and a key-value store. For many APIs it replaces a message broker and a separate store with one component that is simpler to operate. Kipchak already has drivers for RabbitMQ and Kafka; the NATS driver is the lighter alternative.
The driver covers the parts of NATS that an API uses:
- Messaging: publishing to subjects, subscriptions with wildcards and queue groups, and request and reply. A request to a subject that no service is listening on fails immediately instead of waiting for its timeout.
- JetStream: publishing that waits until the stream has stored the message. Each message carries an identifier, so a retried publish is stored once.
- Streams and consumers declared in configuration, and consumers that process messages in a worker loop, acknowledge each one, retry failures with an increasing delay and stop cleanly on a signal.
- Key-value buckets, with updates conditional on a revision and expiry per bucket. They are stored in the same format as the official NATS clients use, so the
natscommand-line tool and services in other languages read the same keys.
The driver uses the wire protocol implementation of the basis-company/nats library, and replaces that library’s connection handling with its own. The library does not verify TLS certificates and, by default, retries a lost connection indefinitely, which makes a request wait for as long as NATS is unavailable. The driver verifies the server’s certificate and host name, supports client certificates, NKeys and credentials files, fails over between servers, and makes a bounded number of reconnection attempts.
Each FrankenPHP worker keeps its connection open between requests. When NATS restarts, the driver reconnects and restores its subscriptions; a request that was in progress fails with a connection error rather than waiting for its timeout, and a JetStream publish is retried once, which is safe because the stream discards the duplicate. In testing, three restarts of NATS during roughly 22,000 requests caused one failed request.
composer require kipchak/driver-nats
The NATS driver documentation describes the configuration, and the starter project now includes an example.
PII Redaction
Personal data reaches places it should not in two common ways: a JSON response that includes fields the client does not need, and a log record that includes a request body, an email address in an error message, or a token in a header. The PII Redaction middleware addresses both from one configuration file, so that no controller has to remember to do it.
- Responses: fields selected by name or by path are removed, masked, shortened, replaced or hashed. A card number becomes
************4242and an email addressj***@example.com. Responses with nothing to redact are sent unchanged. - Logs: every record written through the Kipchak Logger driver, by controllers, drivers or middlewares, is redacted before it is written, including exceptions attached to it.
- Detectors find email addresses, payment card numbers (validated with the Luhn check), IBANs (validated with their checksum), US social security numbers, UK National Insurance numbers, international telephone numbers, JSON Web Tokens and bearer tokens inside any text. IP address detection can be enabled, and an API can add its own patterns.
- Pseudonyms: the
hashaction replaces a value with a keyed HMAC, which is the same for the same value. Support staff can follow one customer through the logs without seeing who the customer is. - Exemptions: named consumers, such as an internal support application, can receive responses unredacted. Logs are always redacted.
Passwords, secrets, tokens, API keys, card numbers and security codes, national identity numbers, passport numbers, IBANs and dates of birth are redacted with no configuration at all.
Rules are compiled once per worker. A 15 KB response of 100 records is redacted in about 0.3 milliseconds with the default rules. In testing, 6,000 concurrent requests returned no personal data in any redacted response, and memory use remained constant.
composer require kipchak/middleware-redaction
The PII Redaction documentation lists the rules, actions and detectors.
Audit Log
SOC 2, ISO 27001, HIPAA and PCI DSS each require a record of who changed what, when, from where, and whether the change succeeded. The Audit Log middleware writes that record for every POST, PUT, PATCH and DELETE request, and for any read that a route chooses to record, such as viewing a patient’s file.
- The identity is the verified one. The consumer and tenant come from Kipchak Identity after authentication, so a record names the client to which the token was issued. Requests refused with 401 or 403 are recorded as well.
- The action is described in the application’s terms. A controller can add the action, such as
order.refund, the target, such as order 42, and further details. The method, route, route arguments, query and body are recorded automatically. - Personal data is redacted with the PII Redaction rules before a record is written.
- Records are tamper-evident. Each record can carry an HMAC-SHA256 signature over its contents, which anyone holding the key can verify. A changed or removed field invalidates the signature.
- Records are delivered where a security team reads them: Kafka, OpenSearch, NATS JetStream and the log, to several at once if required. Each record has a time-ordered identifier, which each destination uses to store a retried record once.
A destination that fails never fails the request. The record is written to the log in full, so that it can be found and replayed, and the destination is skipped for a short period so that an unavailable service does not add a timeout to every request. In testing, 2,400 concurrent requests produced 2,400 records in each of Kafka, OpenSearch and NATS, each stored exactly once and each with a valid signature. With OpenSearch stopped during the test, requests continued at full speed and its records were written to the log.
composer require kipchak/middleware-audit
The Audit Log documentation describes the record, the destinations and how to verify signatures.
Two related improvements
Building these middlewares required two changes to existing packages, both released at the same time.
The Error middleware, version 1.29, can now write exceptions through the Logger driver instead of PHP’s error_log(), by setting 'logExceptionsTo' => 'logger' in kipchak.api.php. Exceptions then share the API’s log format and pass through PII Redaction. Without the setting, the behaviour is unchanged.
The OpenSearch driver, version 1.13, retries a failed OpenSearch node after one second, doubling to at most 30 seconds. The OpenSearch client’s default waited 120 seconds after the first failure, doubling to an hour, and in a long-lived worker that state lasted for the life of the worker. In testing, the driver used OpenSearch again eight seconds after it restarted.