Kipchak now has an OpenTelemetry driver and middleware, both part of Kipchak Enterprise. Together they send an API’s traces, metrics and logs to an OpenTelemetry Collector, and from there to any backend that accepts the OpenTelemetry protocol, such as Grafana Tempo, Jaeger, Honeycomb, Datadog or Elastic.
What is recorded
The middleware gives every request a server span, named after the route it matched, such as
GET /v1/orders/{id}. Everything the request does is recorded inside that span, without changes to the API’s code:
- every request made through the HTTP driver, including the requests made by the OPA, Vault, ClickHouse and OAuth 2 Token drivers;
- every Doctrine query, with its SQL text but never its parameter values;
- every Valkey command, with the command name but never its keys or values;
- every Kafka message sent or received.
Outgoing HTTP requests and Kafka messages carry the trace with them, so a trace follows a call from one Kipchak API into the next, and from a producer to the consumer of its message. Every log line written through the Logger driver carries the identifier of its trace, so the log lines of a slow or failed request can be found from its trace, and the trace from a log line. Request durations, outgoing call durations and database operation durations are recorded as metrics, and an API can record spans and metrics of its own.
Built for worker mode
A FrankenPHP worker handles one request after another and runs nothing in between. The driver therefore exports
after each response has been sent, through a new onTerminate() hook in Kipchak’s core, and writes each export to
the collector without waiting for its reply. Exporting never delays a response. A busy worker sends its telemetry
in batches, and a worker that has become idle sends everything it holds.
If the collector cannot be reached, each worker pauses its exports for a set period and discards the telemetry produced in that time rather than holding it in memory. In testing on the Kipchak Enterprise runtime image, with the collector stopped under load, every request succeeded, throughput fell by 10 to 15%, each worker logged one warning per period, and exports resumed when the collector returned.
Measured cost
A span costs about 15 microseconds to create and export. In testing, a route that wrote and read a database row, set and read a Valkey key, called a second API and wrote a log line took 957 microseconds without telemetry and 1,234 microseconds with every span recorded and logs exported. With 10% of traces sampled, it took 1,085 microseconds. A route that spends its time on real database and network work sees a much smaller share. Over 20,500 requests to that route across two APIs, every span, log line and metric reached the collector, and the memory of each worker remained constant.
composer require kipchak/driver-opentelemetry kipchak/middleware-opentelemetry
The driver documentation describes the configuration, what each driver records and how exports behave, and the middleware documentation describes the server span.
Related improvements
Five existing packages were updated for this release:
- Kipchak core 3.15 adds
onTerminate()andterminate(), for work that should run after the response has been sent. The starter project, version 3.19, callsterminate()in its worker loop; existing APIs add one line to theirs. - The Doctrine driver 1.51 accepts DBAL middleware that applies to every connection it creates.
- The Valkey driver 1.34 lets a package choose the class its pools connect with.
- The Kafka driver 1.16 lets a package observe, and add headers to, every message it sends and receives.
None of these changes the behaviour of an API until a package uses it.