<SYSTEM>This is the full developer documentation for oryn</SYSTEM>

# oryn

> A modular Go framework built on fx. OpenTelemetry native. Documented for humans and agents.

## What is oryn?

oryn is a Go framework to build services, such as HTTP, gRPC or MCP APIs, from small modules glued together by [fx](https://uber-go.github.io/fx/) dependency injection.

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    httpserver.Module,
    httpserver.AsHandler("GET /hello/{name}", func(w http.ResponseWriter, r *http.Request) {
      fmt.Fprintf(w, "hello %s", r.PathValue("name"))
    }),
  ).Run(httpserver.Serve())
}
```

## Modular

An application is the [core](/oryn/core/application/) module plus the modules it needs. Each module is a Go module of the repository, versioned and released on its own:

| Module                                   | What it does                                                                      |
| ---------------------------------------- | --------------------------------------------------------------------------------- |
| [core](/oryn/core/application/)          | Runtime, configuration, OpenTelemetry, logging, health checks, test helpers       |
| [httpserver](/oryn/httpserver/overview/) | HTTP server on `net/http`, with handlers, middlewares and handler groups          |
| [grpcserver](/oryn/grpcserver/overview/) | gRPC server, with services, interceptors, health and reflection                   |
| [mcpserver](/oryn/mcpserver/overview/)   | Model Context Protocol server, exposing tools, resources and prompts to AI agents |
| [sql](/oryn/sql/overview/)               | `database/sql` pool to PostgreSQL, MySQL or SQLite, with migrations               |
| [httpclient](/oryn/httpclient/overview/) | Instrumented HTTP client, with transport middlewares                              |
| [grpcclient](/oryn/grpcclient/overview/) | Instrumented gRPC client connections, configured by name                          |
| [mcpclient](/oryn/mcpclient/overview/)   | MCP client sessions, to call the tools of other MCP servers                       |
| [validator](/oryn/validator/overview/)   | Struct and field validation, with custom validations                              |
| [worker](/oryn/worker/overview/)         | Long-running background workers, with restarts and health                         |
| [cron](/oryn/cron/overview/)             | Scheduled jobs                                                                    |

Modules are extended the same way everywhere: you register your own handlers, middlewares, probes or options with `AsXxx` helpers, and fx builds them with their dependencies. There is no global state to set up and no registration order to get right.

To see them working together, run the [demo application](/oryn/demo/overview/): one tasks module served over HTTP, gRPC and MCP, in one process.

## OpenTelemetry instrumented

Traces, metrics and logs are built in, not bolted on:

* every module is instrumented: an HTTP request, a gRPC call, an MCP tool call or a SQL query gets a span, metrics and a correlated log, with no code,
* the standard `OTEL_*` environment variables work as usual, and always win over the config, so oryn fits any OpenTelemetry deployment,
* in tests, all telemetry is exported in memory, and asserted with the `oteltest` helpers.

See [OpenTelemetry](/oryn/core/opentelemetry/).

## Made for humans and agents

This documentation is written as plain Markdown files in the [`docs/`](https://github.com/go-oryn/oryn/tree/main/docs) directory of the repository: the website you are reading is built from them.

* **Humans** read this website.
* **Agents** (Claude Code, Cursor, Codex…) read the same content as Markdown: the `docs/` files, or [`llms.txt`](https://go-oryn.github.io/oryn/llms.txt) and [`llms-full.txt`](https://go-oryn.github.io/oryn/llms-full.txt) on this website.

The framework itself is designed to be easy for agents to use: one way to do each thing, explicit registrations, typed configuration keys documented in one place, and fast in-memory tests to check the result. See [For agents](/oryn/agents/).

# For agents

> How coding agents can read this documentation, and the rules to follow when writing oryn code.

This documentation is written for coding agents first: every page is a plain Markdown file, and the website is built from these files.

## Read the documentation

| Source                                                                      | Use it when                                                |
| --------------------------------------------------------------------------- | ---------------------------------------------------------- |
| [`llms.txt`](https://go-oryn.github.io/oryn/llms.txt)                       | You need the list of pages, with a description of each     |
| [`llms-full.txt`](https://go-oryn.github.io/oryn/llms-full.txt)             | You want the whole documentation in one file               |
| [`llms-small.txt`](https://go-oryn.github.io/oryn/llms-small.txt)           | You want a shorter version of it                           |
| [`docs/`](https://github.com/go-oryn/oryn/tree/main/docs) in the repository | You work in a checkout of the repository, or need one page |
| [pkg.go.dev](https://pkg.go.dev/github.com/go-oryn/oryn/core)               | You need the API reference of a package                    |

To make a project built with oryn aware of it, add this to its `AGENTS.md` or `CLAUDE.md`:

```markdown
This project uses the oryn Go framework. Read https://go-oryn.github.io/oryn/llms.txt
before writing code, and follow the conventions of the oryn documentation.
```

## Rules for writing oryn code

* **Assemble the application with modules.** Start from `core.New(...)`, add the modules you need (`httpserver.Module`…), and group the application code in your own `fx.Options` modules.
* **Register, do not wire.** Use the `AsXxx` helpers to add handlers, services, tools, middlewares, probes, log handlers, config options or OpenTelemetry options: `httpserver.AsHandler`, `grpcserver.AsService`, `mcpserver.AsCapability`, `worker.AsWorker`, `cron.AsJob`, `validator.AsValidation`, `grpcclient.AsClient`, `httpclient.AsMiddleware`, `health.AsProbe`, `log.AsHandler`, `config.AsOptions`…
* **Inject dependencies in constructors.** A handler, middleware or probe can be given as a constructor: its parameters (`*config.Config`, `*slog.Logger`, `otel.Telemetry`, your services…) are injected by fx.
* **Start servers explicitly.** Modules never start anything by themselves: pass `httpserver.Serve()`, `grpcserver.Serve()`, `mcpserver.Serve()`, `health.Serve()`, `worker.Run()` or `cron.Run()` to `Run`, and one-shot work like `sql.Migrate()` to `RunOnce`.
* **Keep the APIs thin.** Put the business rules and their telemetry in a domain module, and expose it through HTTP handlers, gRPC services or MCP tools that only map requests and errors, like the [demo](/oryn/demo/overview/).
* **Configure with keys, not code.** Configuration keys are namespaced by module (`httpserver.address`, `log.level`…) and documented on each module page. Put them in YAML files, per environment.
* **Call other services with the client modules.** Use the `*http.Client` of `httpclient`, the connections of `grpcclient` and the sessions of `mcpclient`: they are instrumented and propagate the trace context.
* **Do not configure OpenTelemetry globally.** Use the providers from the container, and let the standard `OTEL_*` variables configure the exporters.
* **Log with the context.** Use the `...Context` methods of `*slog.Logger` (`InfoContext`…) so logs are correlated with the active span.
* **Test with coretest.** Start the application with `coretest.Start`, call it in memory (`httptest`, `grpcservertest.Dial`, `mcpservertest.Connect`), serve the services it calls in memory (`httpclienttest.Serve`, `grpcclienttest.Serve`, `mcpclienttest.Serve`), run jobs and workers on demand (`crontest.RunJob`, `workertest.Run`), and assert telemetry with the `oteltest` packages. Tests run in memory: no network, no collector.

## Check your work

```shell
go build ./...
go test -race ./...
```

A test asserting the span or the log of the code you added (see [Testing](/oryn/core/testing/)) proves both the behaviour and the instrumentation.

# Application

> How an oryn application is created, assembled from modules, and run.

The `core` module is the foundation of every oryn application: the [fx](https://uber-go.github.io/fx/) runtime, [configuration](/oryn/core/configuration/), [OpenTelemetry and logging](/oryn/core/opentelemetry/), and [health checks](/oryn/core/health/).

```shell
go get github.com/go-oryn/oryn/core
```

## Create and run an application

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    httpserver.Module,
    users.Module,
  ).Run(httpserver.Serve())
}
```

* `core.New(options...)` returns a `core.App` made of the core module and the given [fx options](https://pkg.go.dev/go.uber.org/fx#Option): modules, constructors (`fx.Provide`), registrations (`httpserver.AsHandler`…).

* `core.WithConfigFS(fsys)` loads the configuration files of `fsys`, see [Configuration](/oryn/core/configuration/).

* `Run(options...)` starts the application with the extra options, blocks until `SIGINT` or `SIGTERM`, then stops it gracefully. It exits the process if the application fails to start.

* `RunOnce(options...)` starts the application then stops it right away, for one-shot commands whose work is done by start hooks, like `sql.Migrate()`. It exits the process on failure.

Options passed to `Run` are the things that run with the application, like servers: `httpserver.Serve()`, `health.Serve()`. Modules never start anything by themselves, so tests can use the same modules without opening ports.

`core.App` is immutable: `WithOptions(options...)` and `WithContext(ctx)` return a copy, so an `App` can be shared, for example between `main` and the tests.

| Method                    | Description                                                               |
| ------------------------- | ------------------------------------------------------------------------- |
| `New(options...)`         | Creates the application                                                   |
| `WithOptions(options...)` | Returns a copy with more fx options                                       |
| `WithContext(ctx)`        | Returns a copy using `ctx` as base context, injected as `context.Context` |
| `Build(options...)`       | Returns the `*fx.App`, without starting it                                |
| `Run(options...)`         | Builds, starts, and waits for a shutdown signal                           |
| `RunOnce(options...)`     | Builds, starts, then stops right away                                     |
| `Options(options...)`     | Returns all the fx options, to build an `fx.App` yourself                 |

## Several entrypoints

Share one `App` between several commands, each running it with its own options. For example, with one `main` package per command in `cmd/`:

```go
// app.go: the application, shared by the commands and the tests.
var App = core.New(core.WithConfigFS(configs.FS), httpserver.Module, sql.Module, sql.AsMigrations(migrations.FS), users.Module)
```

```go
// cmd/serve/main.go: go run ./cmd/serve
func main() {
  app.App.Run(health.Serve(), httpserver.Serve())
}
```

```go
// cmd/migrate/main.go: go run ./cmd/migrate
func main() {
  app.App.RunOnce(sql.Migrate())
}
```

The [demo application](/oryn/demo/overview/) is organized this way.

## Organize the code in modules

Group the code of a feature in a module: a `fx.Options` (or `fx.Module`) value with its constructors and registrations.

users/module.go

```go
package users


var Module = fx.Options(
  fx.Provide(NewRepository),
  httpserver.AsHandlerGroup("/users",
    httpserver.WithHandler("GET /{id}", NewGetUserHandler),
    httpserver.WithHandler("POST /{$}", NewCreateUserHandler),
  ),
  health.AsProbe(NewRepositoryProbe, health.Readiness),
)
```

Extension points always follow the same pattern: an `AsXxx` helper registers a value, or a constructor whose parameters are injected by fx. Registrations are independent from each other: their order only matters where the documentation says so (middlewares, config options), and it is then the registration order.

## Provided dependencies

The core module provides, ready to be injected in any constructor:

| Type                                                                 | Package           | What it is                                      |
| -------------------------------------------------------------------- | ----------------- | ----------------------------------------------- |
| `*config.Config`                                                     | `core/config`     | The configuration                               |
| `*slog.Logger`                                                       | `log/slog`        | The application logger, sent to OpenTelemetry   |
| `otel.Telemetry`                                                     | `core/otel`       | The logger, tracer and meter, in one dependency |
| `trace.Tracer`, `metric.Meter`                                       | OpenTelemetry API | Named after `app.name`                          |
| `trace.TracerProvider`, `metric.MeterProvider`, `log.LoggerProvider` | OpenTelemetry API | For instrumentation libraries                   |
| `propagation.TextMapPropagator`                                      | OpenTelemetry API | Context propagation                             |
| `*resource.Resource`                                                 | OpenTelemetry SDK | The resource describing the application         |
| `*health.Checker`                                                    | `core/health`     | Runs the health probes                          |
| `context.Context`                                                    | `context`         | The base context, see `WithContext`             |

```go
type GreetService struct {
  telemetry otel.Telemetry
}


func NewGreetService(telemetry otel.Telemetry) *GreetService {
  return &GreetService{telemetry: telemetry}
}


func (s *GreetService) Greet(ctx context.Context, name string) string {
  ctx, span := s.telemetry.Tracer().Start(ctx, "greet")
  defer span.End()


  s.telemetry.Logger().InfoContext(ctx, "greeting", "name", name)


  return "hello " + name
}
```

At `debug` [log level](/oryn/core/opentelemetry/#logging), the fx events (provided constructors, invoked functions, lifecycle hooks) are logged too.

# Configuration

> Configuration files, environments, environment variables and config options.

The configuration is a `*config.Config`, built on [viper](https://github.com/spf13/viper), and injected in any constructor.

```go
func NewClient(cfg *config.Config) *Client {
  return &Client{
    url:     cfg.GetString("client.url"),
    timeout: cfg.GetDurationOrDefault("client.timeout", 5*time.Second),
  }
}
```

Besides the viper getters (`GetString`, `GetInt`, `GetDuration`…), `GetXxxOrDefault` return a default when the key is not set, and `AppName()`, `AppVersion()`, `Env()` and `IsTestEnv()` give the application identity.

## Sources

Configuration comes from, lowest to highest precedence:

1. built-in defaults (`app.name: app`, `app.version: 0.0.0`),
2. YAML or JSON files at the root of the config file systems, in lexical order,
3. the files of the sub directory named after the environment, e.g. `prod/`,
4. environment variables prefixed with `APP_`: `httpserver.address` is read from `APP_HTTPSERVER_ADDRESS`,
5. values set with `config.WithValues`.

The environment comes from the `APP_ENV` variable (`dev`, `prod`…). Tests run in the `test` environment.

```plaintext
configs/
├── app.yaml        # loaded in every environment
├── otel.yaml
├── dev/otel.yaml   # loaded when APP_ENV=dev, on top of the root files
└── test/app.yaml   # loaded in tests
```

Embed the files from inside the directory, so they are at the root of the file system:

configs/embed.go

```go
package configs


import "embed"


//go:embed *.yaml */*.yaml
var FS embed.FS
```

```go
core.New(core.WithConfigFS(configs.FS))
```

## Environment variable placeholders

Files can reference environment variables with `${VAR}`, or `${VAR:default}` to use a default when the variable is not set or empty:

```yaml
database:
  url: ${DATABASE_URL}
  pool_size: ${DATABASE_POOL_SIZE:10}
```

## Config options

Modules and applications register config options with `config.AsOptions`:

```go
config.AsOptions(
  config.WithFS(extraFS),
  config.WithValues(map[string]any{"log.level": "debug"}),
)
```

| Option                  | Description                                                                                 |
| ----------------------- | ------------------------------------------------------------------------------------------- |
| `WithFS(fsys)`          | Loads the files of a file system. Later file systems override earlier ones                  |
| `WithValues(values)`    | Sets values, with the highest precedence. Keys can be nested maps or dotted (`"log.level"`) |
| `WithEnvironment(env)`  | Sets the environment, instead of `APP_ENV`                                                  |
| `WithEnvPrefix(prefix)` | Reads `<PREFIX>_ENV` and `<PREFIX>_*` variables instead of `APP_`                           |

Options apply in registration order: when two set the same key, the last registered wins. This is how tests override the application configuration, see [Testing](/oryn/core/testing/).

## Keys

Keys are namespaced by module. The core keys, with their defaults:

```yaml
app:
  name: app                     # service name, OTEL_SERVICE_NAME wins
  version: 0.0.0                # service version


log:
  level: info                   # debug, info, warn or error
  source: false                 # add the source code location
  set_default: true             # set the logger as slog.Default
  console:
    enabled: false              # also print the logs on stdout
    format: text                # text or json


otel:
  disabled: false               # disable the SDK, OTEL_SDK_DISABLED wins
  propagators: [tracecontext, baggage]
  resource:
    attributes: {}              # extra resource attributes
  traces:
    exporter: otlp              # otlp, console, memory or none
    sampler: parentbased_always_on
    sampler_arg: 1
  metrics:
    exporter: otlp
    interval: 60s               # export interval
    runtime: true               # Go runtime metrics, false in tests
  logs:
    exporter: otlp
  exporter:
    otlp:
      protocol: http/protobuf   # http/protobuf or grpc
      endpoint: http://localhost:4318
      headers: {}


health:
  timeout: 5s                   # max duration of a check
  server:
    address: ":8081"            # address of health.Serve()
```

A default applies when a key is not set, or set to an empty value (like a placeholder expanding to nothing). An explicit `0` is kept: for durations and limits, it usually means “none” or “unlimited”, see each key.

Every key can be set with an environment variable: `APP_` followed by the key in upper case, with `_` instead of `.`, for example `APP_LOG_LEVEL=debug`. The standard `OTEL_*` variables win over the `otel.*` keys.

Details: [OpenTelemetry](/oryn/core/opentelemetry/#standard-environment-variables-win), [Logging](/oryn/core/opentelemetry/#logging), [Health checks](/oryn/core/health/).

The other modules document their keys on their Configuration page:

| Keys           | Page                                                         |
| -------------- | ------------------------------------------------------------ |
| `httpserver.*` | [HTTP server configuration](/oryn/httpserver/configuration/) |
| `grpcserver.*` | [gRPC server configuration](/oryn/grpcserver/configuration/) |
| `mcpserver.*`  | [MCP server configuration](/oryn/mcpserver/configuration/)   |
| `httpclient.*` | [HTTP client configuration](/oryn/httpclient/configuration/) |
| `grpcclient.*` | [gRPC client configuration](/oryn/grpcclient/configuration/) |
| `mcpclient.*`  | [MCP client configuration](/oryn/mcpclient/configuration/)   |
| `sql.*`        | [SQL configuration](/oryn/sql/configuration/)                |
| `validator.*`  | [Validator configuration](/oryn/validator/configuration/)    |
| `worker.*`     | [Worker configuration](/oryn/worker/configuration/)          |
| `cron.*`       | [Cron configuration](/oryn/cron/configuration/)              |

# Health checks

> Liveness, readiness and startup probes, and the health HTTP server.

The health checker runs probes for three kinds of check, matching the [Kubernetes probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/):

| Kind               | Endpoint    | Question                                                         |
| ------------------ | ----------- | ---------------------------------------------------------------- |
| `health.Liveness`  | `/livez`    | Is the application alive? A failure means it should be restarted |
| `health.Readiness` | `/readyz`   | Can the application receive traffic?                             |
| `health.Startup`   | `/startupz` | Has the application started?                                     |

## Probes

A probe has a name, and checks one part of the application:

```go
type DatabaseProbe struct {
  db *sql.DB
}


func NewDatabaseProbe(db *sql.DB) *DatabaseProbe {
  return &DatabaseProbe{db: db}
}


func (p *DatabaseProbe) Name() string {
  return "database"
}


func (p *DatabaseProbe) Check(ctx context.Context) error {
  return p.db.PingContext(ctx)
}
```

Register it with `health.AsProbe`, for some kinds of check, or all of them when none is given:

```go
health.AsProbe(NewDatabaseProbe, health.Readiness, health.Startup)
```

Probes run concurrently, and the whole check must finish within `health.timeout`. A panicking probe fails, without crashing the application. The readiness check fails as soon as the application starts shutting down, so load balancers stop sending traffic before the servers stop.

## Health server

`health.Serve()` starts an HTTP server with the application, exposing the three endpoints. They answer `200` when healthy and `503` otherwise, with the probe results as JSON when the `verbose` query parameter is present:

```shell
curl 'localhost:8081/readyz?verbose'
```

```go
core.New(
  core.WithConfigFS(configs.FS),
  health.AsProbe(NewDatabaseProbe, health.Readiness),
).Run(health.Serve())
```

| Config key              | Default |
| ----------------------- | ------- |
| `health.server.address` | `:8081` |
| `health.timeout`        | `5s`    |

To check health in code, inject the `*health.Checker` and call `Check(ctx, kind)`.

# OpenTelemetry

> Traces, metrics and logs, the slog logger, exporters, sampling, and how the OTEL_* environment variables apply.

oryn sets up the OpenTelemetry SDK for the three signals: traces, metrics and logs. The modules use it to instrument what they do, and your code uses it through the injected `otel.Telemetry`, `trace.Tracer`, `metric.Meter` and `*slog.Logger`.

```go
func NewCheckoutService(telemetry otel.Telemetry) (*CheckoutService, error) {
  orders, err := telemetry.Meter().Int64Counter("checkout.orders")
  if err != nil {
    return nil, err
  }


  return &CheckoutService{telemetry: telemetry, orders: orders}, nil
}


func (s *CheckoutService) Checkout(ctx context.Context, cart Cart) error {
  ctx, span := s.telemetry.Tracer().Start(ctx, "checkout")
  defer span.End()


  s.orders.Add(ctx, 1)
  s.telemetry.Logger().InfoContext(ctx, "checked out", "items", len(cart.Items))


  return nil
}
```

The application tracer, meter and logger are named after `app.name`. Instrumentation libraries get the providers (`trace.TracerProvider`, `metric.MeterProvider`, `log.LoggerProvider`) and the `propagation.TextMapPropagator` from the container, and use their own instrumentation scope. oryn also sets them as the OpenTelemetry globals, for third party libraries that only use those.

## Logging

The application logger is a standard `*slog.Logger`. It sends records to the OpenTelemetry logger provider through the [otelslog](https://pkg.go.dev/go.opentelemetry.io/contrib/bridges/otelslog) bridge, so logs are exported with the traces and metrics.

```go
func NewGreeter(logger *slog.Logger) *Greeter {
  return &Greeter{logger: logger}
}


func (g *Greeter) Greet(ctx context.Context, name string) {
  g.logger.InfoContext(ctx, "greeting", "name", name)
}
```

Use the `...Context` methods (`InfoContext`, `ErrorContext`…): the record then gets the trace and span IDs of the active span, so logs and traces are linked in your observability backend.

| Config key            | Default | Description                                                               |
| --------------------- | ------- | ------------------------------------------------------------------------- |
| `log.level`           | `info`  | `debug`, `info`, `warn` or `error`, with an optional offset like `info+2` |
| `log.source`          | `false` | Add the source code location                                              |
| `log.console.enabled` | `false` | Also print records on stdout                                              |
| `log.console.format`  | `text`  | `text` or `json`                                                          |
| `log.set_default`     | `true`  | Set the logger as `slog.Default()`                                        |

At `debug` level, the fx events (provided constructors, invoked functions, lifecycle hooks) are logged too.

Register an extra `slog.Handler`, receiving all the records at or above `log.level`, with `log.AsHandler`:

```go
log.AsHandler(func() slog.Handler {
  return slog.NewJSONHandler(os.Stderr, nil)
})
```

## Standard environment variables win

oryn follows the [OpenTelemetry environment variables specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/). **A standard `OTEL_*` variable always wins over the equivalent config key**, so an application deployed with the usual OpenTelemetry settings (Kubernetes operator, collector sidecar…) needs no config change.

| Config key                         | Environment variable          | Default                   |
| ---------------------------------- | ----------------------------- | ------------------------- |
| `app.name`                         | `OTEL_SERVICE_NAME`           | `app`                     |
| `app.version`                      |                               | `0.0.0`                   |
| `otel.resource.attributes` (map)   | `OTEL_RESOURCE_ATTRIBUTES`    |                           |
| `otel.disabled`                    | `OTEL_SDK_DISABLED`           | `false`                   |
| `otel.propagators` (list)          | `OTEL_PROPAGATORS`            | `tracecontext,baggage`    |
| `otel.traces.exporter`             | `OTEL_TRACES_EXPORTER`        | `otlp`                    |
| `otel.traces.sampler`              | `OTEL_TRACES_SAMPLER`         | `parentbased_always_on`   |
| `otel.traces.sampler_arg`          | `OTEL_TRACES_SAMPLER_ARG`     | `1`                       |
| `otel.metrics.exporter`            | `OTEL_METRICS_EXPORTER`       | `otlp`                    |
| `otel.metrics.interval`            | `OTEL_METRIC_EXPORT_INTERVAL` | `60s`                     |
| `otel.metrics.runtime`             |                               | `true` (`false` in tests) |
| `otel.logs.exporter`               | `OTEL_LOGS_EXPORTER`          | `otlp`                    |
| `otel.exporter.otlp.protocol`      | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf`           |
| `otel.exporter.otlp.endpoint`      | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318`   |
| `otel.exporter.otlp.headers` (map) | `OTEL_EXPORTER_OTLP_HEADERS`  |                           |

* Exporters are `otlp`, `console`, `memory` or `none`. The OTLP protocol is `grpc` or `http/protobuf`.
* Signal specific variables (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`…) and the SDK tuning variables (`OTEL_BSP_*`, `OTEL_BLRP_*`…) work as usual.
* `otel.exporter.otlp.endpoint` is a base URL: like `OTEL_EXPORTER_OTLP_ENDPOINT`, `/v1/<signal>` is appended for HTTP.
* Samplers are `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off` and `parentbased_traceidratio`, with `sampler_arg` as ratio.
* Propagators are those of [autoprop](https://pkg.go.dev/go.opentelemetry.io/contrib/propagators/autoprop): `tracecontext`, `baggage`, `b3`, `b3multi`, `jaeger`, `xray`, `ottrace`.

## Local development

Print the telemetry on the console in the `dev` environment:

configs/dev/otel.yaml

```yaml
otel:
  traces:
    exporter: console
  metrics:
    exporter: none
  logs:
    exporter: console
```

Or send it to a local collector or backend with OTLP, for example the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) or [Jaeger](https://www.jaegertracing.io/), listening on `localhost:4318`.

## Resource

The resource describes the application. Its attributes come, lowest to highest precedence, from:

1. the SDK and the Go runtime,
2. the config: `service.name` (`app.name`), `service.version` (`app.version`), `deployment.environment.name` (the environment), and `otel.resource.attributes`,
3. the options registered with `otel.AsResourceOptions`, for example extra detectors,
4. the `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` environment variables.

## Runtime metrics

Go runtime metrics (memory, garbage collection, goroutines…) are collected by default, except in tests. Disable them with `otel.metrics.runtime: false`.

## Extend the SDK

Register options for the SDK resource and providers:

```go
otel.AsResourceOptions(resource.WithHost(), resource.WithContainer())
otel.AsTracerProviderOptions(sdktrace.WithSpanProcessor(myProcessor))
otel.AsMeterProviderOptions(sdkmetric.WithView(myView))
otel.AsLoggerProviderOptions(sdklog.WithProcessor(myProcessor))
```

## Tests

In the `test` environment, all signals are exported in memory, whatever the configuration, and can be asserted with the `oteltest` packages. See [Testing](/oryn/core/testing/).

# Testing

> Run the application in tests with coretest, and assert traces, metrics and logs in memory.

`coretest` runs an oryn application in tests, in the `test` environment:

* all telemetry is exported in memory, whatever the configuration,
* logs are not printed on the console,
* the logger is not set as `slog.Default()`, so parallel tests do not interfere.

## Start the application

```go
func TestGreet(t *testing.T) {
  t.Parallel()


  var greeter *Greeter


  coretest.Start(t, core.New(greet.Module), fx.Populate(&greeter))


  assert.Equal(t, "hello oryn", greeter.Greet(t.Context(), "oryn"))
}
```

| Function                             | Description                                                                |
| ------------------------------------ | -------------------------------------------------------------------------- |
| `coretest.Start(t, app, options...)` | Builds and starts the application, and stops it when the test ends         |
| `coretest.New(t, app, options...)`   | Builds the application without starting it                                 |
| `coretest.TestOptions()`             | The options added to every test application, to build an `fx.App` yourself |

The options come after the application ones: get dependencies with `fx.Populate`, replace them with `fx.Replace` or `fx.Decorate`, and override the configuration with `config.AsOptions`:

```go
coretest.Start(t, app,
  config.AsOptions(config.WithValues(map[string]any{"log.level": "debug"})),
  fx.Decorate(func() Clock { return fakeClock }),
  fx.Populate(&service),
)
```

The application uses the test context as base context: it is cancelled when the test ends.

## Assert telemetry

The in-memory exporters are in the container. Assert on them with the `otel/oteltest` packages:

```go
func TestCheckout(t *testing.T) {
  t.Parallel()


  var (
    service *CheckoutService
    spans   *tracetest.InMemoryExporter
    logs    *logtest.InMemoryExporter
    metrics *sdkmetric.ManualReader
  )


  coretest.Start(t, app, fx.Populate(&service, &spans, &logs, &metrics))


  require.NoError(t, service.Checkout(t.Context(), cart))


  spantest.HasSpan(t, spans, spantest.WithName("checkout"))
  logtest.HasRecord(t, logs, logtest.WithBody("checked out"), logtest.WithLevel(slog.LevelInfo))
  assert.Equal(t, int64(1), metrictest.Int64Sum(t, metrics, "checkout.orders"))
}
```

| Package      | Assertions                                                | Filters                                                                                                       |
| ------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `spantest`   | `HasSpan`, `HasNoSpan`, `SpanCount`, `Find`               | `WithName`, `WithNameContains`, `WithKind`, `WithStatus`, `WithAttribute`, `WithAttributeValue`, `WithParent` |
| `logtest`    | `HasRecord`, `HasNoRecord`, `RecordCount`                 | `WithBody`, `WithBodyContains`, `WithLevel`, `WithSeverity`, `WithAttribute`, `WithAttributeValue`            |
| `metrictest` | `HasMetric`, `HasNoMetric`, `Int64Sum`, `Collect`, `Find` | `WithName`, `WithUnit`, `WithAttributeValue`                                                                  |

When an assertion fails, the message lists what was recorded, to see what went wrong without debugging.

## Test failures to start

To test that an application fails to start, build it with `fx.New` and check the error:

```go
app := fx.New(core.New(myModule).Options(coretest.TestOptions(), fx.NopLogger)...)
require.ErrorContains(t, app.Err(), "invalid configuration")
```

# Configuration

> Every cron configuration key, with its default and when to change it.

The `cron.*` keys configure the scheduler and override the registered jobs. All of them are optional.

## Reference

```yaml
cron:
  location: UTC               # time zone of the cron expressions
  singleton: true             # skip a run while the previous one of the job is still running
  timeout: 0s                 # max duration of an execution, 0 = none
  log:
    enabled: true             # log every execution
  jobs:
    <name>:                   # the name given to cron.AsJob
      schedule: "0 3 * * *"   # replaces the registered schedule
      enabled: true           # false: never scheduled, still runnable with RunNow
      singleton: true         # cron.singleton for this job
      timeout: 0s             # cron.timeout for this job, 0s = none
```

| Key                          | Type     | Default                                | Environment variable             |
| ---------------------------- | -------- | -------------------------------------- | -------------------------------- |
| `cron.location`              | string   | `UTC`                                  | `APP_CRON_LOCATION`              |
| `cron.singleton`             | bool     | `true`                                 | `APP_CRON_SINGLETON`             |
| `cron.timeout`               | duration | `0s` (none)                            | `APP_CRON_TIMEOUT`               |
| `cron.log.enabled`           | bool     | `true`                                 | `APP_CRON_LOG_ENABLED`           |
| `cron.jobs.<name>.schedule`  | string   | the registered schedule                | `APP_CRON_JOBS_<NAME>_SCHEDULE`  |
| `cron.jobs.<name>.enabled`   | bool     | `true`                                 | `APP_CRON_JOBS_<NAME>_ENABLED`   |
| `cron.jobs.<name>.singleton` | bool     | `WithSingleton`, then `cron.singleton` | `APP_CRON_JOBS_<NAME>_SINGLETON` |
| `cron.jobs.<name>.timeout`   | duration | `WithTimeout`, then `cron.timeout`     | `APP_CRON_JOBS_<NAME>_TIMEOUT`   |

Durations use the Go syntax: `500ms`, `30s`, `2m`. In environment variables, the job name is upper cased: `APP_CRON_JOBS_CLEANUP_SCHEDULE`. Use names made of letters, digits and underscores, so that they can be set from the environment.

For each job, a `cron.jobs.<name>.*` key wins over the job option, which wins over the global key.

## Time zone

Cron expressions are evaluated in `cron.location`, an [IANA time zone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) name, `UTC` by default. `Local` is the time zone of the host:

```yaml
cron:
  location: Europe/Paris
```

Prefer UTC for jobs scheduled at night: with a time zone observing daylight saving time, the hours around the change are skipped or repeated once a year. An unknown time zone fails the application start. `@every` intervals do not depend on the time zone.

## Schedules

Change the schedule of a job without a new build, for example per environment, or disable it:

```yaml
cron:
  jobs:
    cleanup:
      schedule: "@every 1m"
    report:
      enabled: false
```

A disabled job is not scheduled, but stays listed by `Scheduler.Jobs`, and can still be run with `Scheduler.RunNow`. An invalid schedule fails the application start. See [Schedules](/oryn/cron/overview/#schedules) for the syntax.

## Overlaps

A run is skipped while the previous one of the same job is still running. Allow overlapping runs for every job, or a single one:

```yaml
cron:
  singleton: false
  jobs:
    cleanup:
      singleton: true
```

## Timeouts

Jobs have no timeout by default. Set one for every job, and override it per job:

```yaml
cron:
  timeout: 5m
  jobs:
    report:
      timeout: 30m
```

The job context is canceled once the timeout expires. A `0s` job key removes the timeout for that job, whatever the job option and the global key.

## Execution logs

Every execution is logged, see [Observability](/oryn/cron/observability/#logs). Disable the logs when the traces are enough, or for jobs running every few seconds:

```yaml
cron:
  log:
    enabled: false
```

Panics are logged even when the execution logs are disabled.

## Per environment

configs/app.yaml

```yaml
cron:
  jobs:
    cleanup:
      schedule: "0 3 * * *"


# configs/dev/cron.yaml (APP_ENV=dev)
cron:
  jobs:
    cleanup:
      schedule: "@every 1m"
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the cron job executions.

Every execution of a job, scheduled or run on demand, is traced, measured and logged with the application providers, under the `github.com/go-oryn/oryn/cron` instrumentation scope. A panic in a job is recovered, and recorded as a failure.

## Traces

Each execution gets an internal span:

* named `cron <name>`: `cron cleanup`,
* with the `cron.job.name`, `cron.job.schedule`, `cron.job.trigger` and `cron.job.outcome` attributes (see below),
* with an error status and the recorded error when the job fails.

A scheduled execution starts a new trace. An execution triggered by `Scheduler.RunNow` is a child of the span of its context. Spans created in the job with its context are children of the execution span.

| Attribute           | Values                                                      |
| ------------------- | ----------------------------------------------------------- |
| `cron.job.name`     | `cleanup`                                                   |
| `cron.job.schedule` | `0 3 * * *`                                                 |
| `cron.job.trigger`  | `schedule`, or `manual` for `RunNow`                        |
| `cron.job.outcome`  | `success`, `failure` (the job returned an error) or `panic` |

## Metrics

| Metric              | Type           | Description                                                                            |
| ------------------- | -------------- | -------------------------------------------------------------------------------------- |
| `cron.job.runs`     | counter        | Number of executions, with the `cron.job.name` and `cron.job.outcome` attributes       |
| `cron.job.duration` | histogram, `s` | Duration of the executions, with the `cron.job.name` and `cron.job.outcome` attributes |

Alert on failures with `cron.job.runs{cron.job.outcome!="success"}`, and on a job that stopped running with the absence of increase of `cron.job.runs`.

## Logs

| Message              | Level | Attributes                            |
| -------------------- | ----- | ------------------------------------- |
| `cron job started`   | debug | `job`, `trigger`                      |
| `cron job succeeded` | info  | `job`, `trigger`, `duration`          |
| `cron job failed`    | error | `job`, `trigger`, `duration`, `error` |
| `cron job panic`     | error | `job`, `panic`, `stack`               |

The logs are correlated with the execution span. Disable the execution logs with `cron.log.enabled: false`; `cron job panic` is always logged.

When the scheduler starts, `cron scheduler started` is logged at debug level, with the number of jobs.

# Cron

> The cron module, scheduled jobs registered through fx, run with the application and instrumented with OpenTelemetry.

The `cron` module runs jobs on a schedule, with the application. It is built on [gocron](https://github.com/go-co-op/gocron), and every execution is traced, measured and logged.

```shell
go get github.com/go-oryn/oryn/cron
```

## Quick start

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    cron.Module,
    cron.AsJob("cleanup", "0 3 * * *", NewCleanupJob),
  ).Run(cron.Run())
}
```

* `cron.Module` provides the `*cron.Scheduler`, with the registered jobs.
* `cron.Run()` starts the scheduler with the application, and stops it gracefully when the application stops, see [Stop](#stop). Without it, the jobs only run on demand.
* Every execution is traced, measured and logged, see [Observability](/oryn/cron/observability/).

## Jobs

A job implements `cron.Job`:

```go
type Job interface {
  Run(ctx context.Context) error
}
```

Register it with a unique name and a schedule. The job is either a value, a plain `func(context.Context) error`, or a constructor whose parameters are injected by fx:

```go
type CleanupJob struct {
  repository *Repository
}


func NewCleanupJob(repository *Repository) *CleanupJob {
  return &CleanupJob{repository: repository}
}


func (j *CleanupJob) Run(ctx context.Context) error {
  return j.repository.DeleteExpired(ctx)
}
```

```go
cron.AsJob("cleanup", "0 3 * * *", NewCleanupJob)
cron.AsJob("heartbeat", "@every 30s", func(ctx context.Context) error { ... })
```

The context of a job carries the span of the execution, and is canceled when the job times out, or when the application stop times out. A returned error or a panic marks the execution as failed: it is recorded, and the job runs again at its next schedule.

The application fails to start if a name is empty or registered twice, if a job is not a `cron.Job` or a constructor returning one, or if a schedule is invalid.

## Schedules

| Schedule                                              | Runs                                                                                                 |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `0 3 * * *`                                           | a cron expression, with 5 fields: minute, hour, day of month, month, day of week                     |
| `*/10 * * * * *`                                      | a cron expression, with 6 fields, the first one being the second                                     |
| `@hourly`, `@daily`, `@weekly`, `@monthly`, `@yearly` | a predefined schedule                                                                                |
| `@every 1m30s`                                        | at a fixed interval, any positive Go duration, the first run one interval after the scheduler starts |

Cron expressions are evaluated in the `cron.location` time zone, UTC by default. The configuration can override the schedule of a job, or disable it:

```yaml
cron:
  location: Europe/Paris
  jobs:
    cleanup:
      schedule: "0 4 * * *"
    heartbeat:
      enabled: false
```

See [Configuration](/oryn/cron/configuration/).

## Overlaps

By default, a job does not overlap itself: if it is still running when it is due again, that run is skipped, and the job runs at the following schedule. To allow concurrent runs of a job:

```go
cron.AsJob("poll", "@every 10s", NewPollJob, cron.WithSingleton(false))
```

or for every job, with `cron.singleton: false`.

## Timeouts

Jobs have no timeout by default. Bound an execution with `cron.WithTimeout`, `cron.timeout` for every job, or `cron.jobs.<name>.timeout`:

```go
cron.AsJob("report", "@hourly", NewReportJob, cron.WithTimeout(10*time.Minute))
```

Once the timeout expires, the job context is canceled: the job should return. Its error, usually `context.DeadlineExceeded`, marks the execution as failed.

## Run on demand

The `*cron.Scheduler` runs a job immediately with `RunNow`, whatever its schedule, even if it is disabled or the scheduler is not started. It returns the error of the job, or `cron.ErrJobNotFound`. The execution is instrumented like a scheduled one, as a child of the span of the given context, for example from an admin endpoint:

```go
func (h *AdminHandler) Run(w http.ResponseWriter, r *http.Request) {
  if err := h.scheduler.RunNow(r.Context(), r.PathValue("job")); err != nil {
    ...
  }
}
```

`Jobs` lists the jobs in registration order, with their name, schedule, whether they are enabled, and their next run once the scheduler is started.

## Stop

When the application stops, the scheduler stops scheduling jobs, and waits for the running ones to finish. If the stop timeout expires first, the context of the running jobs is canceled, and the application stop fails with `context.DeadlineExceeded`.

## Configuration

The `cron.*` keys set the time zone, the overlaps, the timeouts and the execution logs, and override the schedule of each job. See [Configuration](/oryn/cron/configuration/).

# Testing

> Run cron jobs on demand in tests, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/), without `cron.Run()`, and run a job immediately with `crontest.RunJob`, without waiting for its schedule. It returns the error of the job:

```go
func TestCleanupJob(t *testing.T) {
  t.Parallel()


  var (
    scheduler *cron.Scheduler
    spans     *tracetest.InMemoryExporter
    metrics   *sdkmetric.ManualReader
  )


  coretest.Start(t, core.New(cron.Module, cleanup.Module), fx.Populate(&scheduler, &spans, &metrics))


  require.NoError(t, crontest.RunJob(t, scheduler, "cleanup"))


  spantest.HasSpan(t, spans,
    spantest.WithName("cron cleanup"),
    spantest.WithAttributeValue(cron.JobOutcomeKey.String(cron.OutcomeSuccess)),
  )
  assert.Equal(t, int64(1), metrictest.Int64Sum(t, metrics, cron.MetricJobRuns,
    cron.JobNameKey.String("cleanup"), cron.JobOutcomeKey.String(cron.OutcomeSuccess)))
}
```

The execution goes through the whole module: configuration, timeout, panic recovery and instrumentation. A disabled job can be run too.

## On a schedule

To test the scheduling itself, start the scheduler with `cron.Run()`, and give the job a short interval with `crontest.WithSchedule`, which overrides `cron.jobs.<name>.schedule`:

```go
coretest.Start(t, app, crontest.WithSchedule("cleanup", "@every 10ms"), cron.Run())


assert.Eventually(t, func() bool { return job.Runs() >= 2 }, time.Second, 5*time.Millisecond)
```

When the test ends, the application stops and waits for the running jobs.

## Replace a dependency

Replace a dependency of the job with `fx.Replace` or `fx.Decorate`, see [Testing](/oryn/core/testing/).

# APIs

> The HTTP, gRPC and MCP APIs of the demo, three thin layers registered on the same tasks service, and how to call them.

Each API is a package of `internal/` with its own fx module: `internal/http` (package `httpapi`), `internal/grpc` (`grpcapi`) and `internal/mcp` (`mcpapi`), named so they do not collide with the `net/http`, gRPC and MCP SDK packages they import. It depends on the `*tasks.Service`, maps the requests to it, and maps the `tasks.ErrNotFound` and `tasks.ErrInvalidTitle` errors to its protocol:

| Error                   | HTTP  | gRPC               | MCP        |
| ----------------------- | ----- | ------------------ | ---------- |
| `tasks.ErrInvalidTitle` | `400` | `INVALID_ARGUMENT` | tool error |
| `tasks.ErrNotFound`     | `404` | `NOT_FOUND`        | tool error |
| other                   | `500` | `INTERNAL`         | tool error |

## HTTP

internal/http/module.go

```go
var Module = fx.Module("httpapi",
  httpserver.AsMiddleware(TraceIDMiddleware),
  httpserver.AsHandlerGroup("/api",
    httpserver.WithMiddlewares(JSONMiddleware),
    httpserver.WithHandler("GET /tasks", NewListHandler),
    httpserver.WithHandler("POST /tasks", NewCreateHandler),
    httpserver.WithHandler("GET /tasks/{id}", NewGetHandler),
    httpserver.WithHandler("POST /tasks/{id}/complete", NewCompleteHandler),
    httpserver.WithHandler("DELETE /tasks/{id}", NewDeleteHandler),
  ),
)


// internal/http/handlers.go
func NewCreateHandler(service *tasks.Service, logger *slog.Logger) http.HandlerFunc {
  return func(w http.ResponseWriter, r *http.Request) {
    var request createRequest
    if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
      writeJSON(w, http.StatusBadRequest, errorBody(`expected a JSON body like {"title": "..."}`))
      return
    }


    task, err := service.Create(r.Context(), request.Title)
    ...
  }
}
```

```shell
curl -X POST localhost:8080/api/tasks -d '{"title": "Try oryn"}'
curl localhost:8080/api/tasks
curl -X POST localhost:8080/api/tasks/<id>/complete
curl -X DELETE localhost:8080/api/tasks/<id>
```

The `TraceIDMiddleware` adds an `X-Trace-Id` header to every response: the ID of the trace of the request. See [HTTP server](/oryn/httpserver/overview/).

## gRPC

The service is defined in `proto/tasks/v1/tasks.proto`. The generated code is committed; `go generate ./proto` regenerates it with [protoc](https://protobuf.dev/installation/) and the plugins declared as `tool` dependencies in `go.mod`.

internal/grpc/service.go

```go
var Module = fx.Module("grpcapi",
  grpcserver.AsService(&tasksv1.TasksService_ServiceDesc, NewTasksService),
)


type TasksService struct {
  tasksv1.UnimplementedTasksServiceServer
  service *tasks.Service
}


func (s *TasksService) CreateTask(ctx context.Context, request *tasksv1.CreateTaskRequest) (*tasksv1.Task, error) {
  task, err := s.service.Create(ctx, request.GetTitle())
  if err != nil {
    return nil, grpcError(err)
  }


  return toProto(task), nil
}
```

Reflection is enabled in `configs/app.yaml` (`grpcserver.reflection.enabled`), so [grpcurl](https://github.com/fullstorydev/grpcurl) needs no proto file:

```shell
grpcurl -plaintext localhost:50051 list
grpcurl -plaintext -d '{"title": "Try oryn"}' localhost:50051 tasks.v1.TasksService/CreateTask
grpcurl -plaintext localhost:50051 tasks.v1.TasksService/ListTasks
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
```

See [gRPC server](/oryn/grpcserver/overview/).

## MCP

The tools are typed: their input and output JSON schemas are inferred from the Go types.

internal/mcp/tools.go

```go
var Module = fx.Module("mcpapi",
  mcpserver.AsCapability(NewTools),
)


type CreateInput struct {
  Title string `json:"title" jsonschema:"the title of the task"`
}


func (t *Tools) Register(server *mcp.Server) error {
  mcp.AddTool(server, &mcp.Tool{Name: "create_task", Description: "Create a task."},
    func(ctx context.Context, _ *mcp.CallToolRequest, input CreateInput) (*mcp.CallToolResult, tasks.Task, error) {
      task, err := t.service.Create(ctx, input.Title)
      return nil, task, err
    })
  // list_tasks, complete_task, delete_task...
  return nil
}
```

Connect an agent, then ask it to manage your tasks:

```shell
claude mcp add --transport http oryn-demo http://localhost:8082/mcp
```

Or explore the tools with the [MCP Inspector](https://github.com/modelcontextprotocol/inspector): `npx @modelcontextprotocol/inspector`, then connect to `http://localhost:8082/mcp` with the streamable HTTP transport. See [MCP server](/oryn/mcpserver/overview/).

# Entrypoints

> How the demo shares one application between several commands in cmd/, one serving HTTP, gRPC and MCP together, one applying the migrations.

The demo defines its application once, in `app.go`, and runs it from one `main` package per command in `cmd/`. Each command passes the options it needs to `Run` or `RunOnce`.

## The application

app.go

```go
package demo


var App = core.New(
  core.WithConfigFS(configs.FS),
  httpserver.Module,
  grpcserver.Module,
  mcpserver.Module,
  sql.Module,
  sql.AsMigrations(migrations.FS),
  // The tasks domain, and the three APIs on it.
  tasks.Module,
  httpapi.Module,
  grpcapi.Module,
  mcpapi.Module,
)
```

Modules only provide and register things: nothing listens on a port and nothing touches the database until a command asks for it.

## serve: all the servers in one process

cmd/serve/main.go

```go
func main() {
  demo.App.Run(
    health.Serve(),
    httpserver.Serve(),
    grpcserver.Serve(),
    mcpserver.Serve(),
  )
}
```

```shell
go run ./cmd/serve
```

| Server        | Address     | Config key                            |
| ------------- | ----------- | ------------------------------------- |
| HTTP API      | `:8080`     | `httpserver.address`                  |
| gRPC API      | `:50051`    | `grpcserver.address`                  |
| MCP           | `:8082/mcp` | `mcpserver.address`, `mcpserver.path` |
| Health checks | `:8081`     | `health.server.address`               |

They start together, share the same `tasks.Service` and database pool, and stop gracefully together on `SIGINT` or `SIGTERM`. If one fails, the whole application stops. To run one API alone, write a command passing only its `Serve()`.

## migrate: a one-shot command

cmd/migrate/main.go

```go
func main() {
  demo.App.RunOnce(sql.Migrate())
}
```

```shell
go run ./cmd/migrate
```

`RunOnce` starts the application, which applies the pending migrations, then stops it. Only the database pool is built: fx builds what the invoked options need, so the servers and handlers are left out. Run it before deploying a new version, as a job or an init container.

## Why not cobra?

A `main` package per command is enough: each one is a few lines, `go run ./cmd/<name>` and `go build ./cmd/...` work out of the box, and there is no flag parsing to maintain, the configuration coming from the [config files and environment](/oryn/core/configuration/). Add a CLI library when a command needs arguments.

## Build

```shell
go build -o bin/ ./cmd/...
bin/migrate && bin/serve
```

# Demo application

> A complete oryn application managing tasks over HTTP, gRPC and MCP, with one domain module, a database, telemetry and in-memory tests.

The [go-oryn/demo](https://github.com/go-oryn/demo) repository is a complete oryn application: it manages tasks over an HTTP API, a gRPC API, and MCP tools for AI agents. The three APIs are thin layers over **one** `tasks` module, so a task created over gRPC is listed over HTTP or by an agent.

```plaintext
            HTTP :8080          gRPC :50051          MCP :8082
                │                    │                    │
          ┌─────▼─────┐        ┌─────▼─────┐        ┌─────▼─────┐
          │   http    │        │   grpc    │        │    mcp    │   internal/: one package per API
          └─────┬─────┘        └─────┬─────┘        └─────┬─────┘
                └────────────────────┼────────────────────┘
                              ┌──────▼──────┐
                              │    tasks    │   Service: rules and telemetry
                              │             │   Store: the Repository, SQL
                              └──────┬──────┘
                              SQLite / PostgreSQL
```

## Run it

```shell
git clone https://github.com/go-oryn/demo.git && cd demo
go run ./cmd/migrate   # create the SQLite database
go run ./cmd/serve     # HTTP, gRPC, MCP and health servers
```

```shell
curl -X POST localhost:8080/api/tasks -d '{"title": "Try oryn"}'
grpcurl -plaintext localhost:50051 tasks.v1.TasksService/ListTasks
claude mcp add --transport http oryn-demo http://localhost:8082/mcp
```

## Layout

| Path                                                | Content                                                                                                               |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `app.go`                                            | The application: the oryn modules and the demo modules, shared by the commands and the tests                          |
| `cmd/serve`, `cmd/migrate`                          | The commands, see [Entrypoints](/oryn/demo/entrypoints/)                                                              |
| `internal/tasks/`                                   | The domain module: `Service`, and its `Store` implemented by the `Repository`, see [Telemetry](/oryn/demo/telemetry/) |
| `internal/http/`, `internal/grpc/`, `internal/mcp/` | The APIs on the `tasks.Service`, see [APIs](/oryn/demo/apis/)                                                         |
| `proto/`                                            | The protobuf definition of the gRPC API, and its generated code                                                       |
| `migrations/`                                       | The SQL migrations, embedded                                                                                          |
| `configs/`                                          | The configuration, per environment                                                                                    |
| `internal/demotest/`                                | Starts the application in tests, with in-memory API clients, see [Testing](/oryn/demo/testing/)                       |
| `compose.yaml`                                      | PostgreSQL and a Grafana observability stack                                                                          |

## What it shows

| Feature                                                                   | Where                                          |
| ------------------------------------------------------------------------- | ---------------------------------------------- |
| One domain module reused by three APIs                                    | `internal/tasks/module.go`, `app.go`           |
| Several commands on one application                                       | `cmd/`, [Entrypoints](/oryn/demo/entrypoints/) |
| HTTP handlers, a handler group and middlewares                            | `internal/http/module.go`                      |
| A gRPC service, with health and reflection                                | `internal/grpc/service.go`                     |
| Typed MCP tools                                                           | `internal/mcp/tools.go`                        |
| Custom spans, metrics and logs, correlated with the API and SQL spans     | `internal/tasks/service.go`                    |
| In-memory tests of the three APIs, asserting telemetry, with `fx.Replace` | `*_test.go`, `internal/demotest/`              |

# Telemetry

> The traces, metrics and logs of the demo, from the HTTP, gRPC and MCP servers down to the SQL queries, and how to see them on stdout or in Grafana.

Every layer of the demo produces telemetry, correlated in the same traces: the API servers and the database pool by the oryn modules, with no code, and the business operations by the `tasks.Service`.

## The tasks service

The service gets the application tracer, meter and logger from the container, as an `otel.Telemetry`:

internal/tasks/service.go

```go
func NewService(store Store, telemetry otel.Telemetry) (*Service, error) {
  created, err := telemetry.Meter().Int64Counter("tasks.created", metric.WithDescription("Number of created tasks"))
  ...
}


func (s *Service) Create(ctx context.Context, title string) (Task, error) {
  ctx, span := s.telemetry.Tracer().Start(ctx, "tasks.create")
  defer span.End()


  task := Task{ID: newID(), Title: strings.TrimSpace(title), CreatedAt: time.Now().UTC()}
  if task.Title == "" {
    return Task{}, s.fail(span, ErrInvalidTitle)
  }


  span.SetAttributes(TaskIDKey.String(task.ID))


  if err := s.store.Insert(ctx, task); err != nil {
    return Task{}, s.fail(span, err)
  }


  s.created.Add(ctx, 1)
  s.telemetry.Logger().InfoContext(ctx, "task created", "task_id", task.ID)


  return task, nil
}
```

* Each operation has a span: `tasks.list`, `tasks.get`, `tasks.create`, `tasks.complete`, `tasks.delete`, with the `task.id` attribute.
* Unexpected errors set the span status to `Error`. Expected ones (not found, invalid title) are only recorded as a `tasks.error` attribute: they are not failures of the service.
* The context carries the span to the store, so the SQL spans are its children, and to the logger, so the logs are correlated with it.

## Traces

The same `tasks.create` span, whatever the API:

```plaintext
POST /api/tasks                         httpserver
└── tasks.create                        tasks.Service
    └── INSERT                          sql


tasks.v1.TasksService/CreateTask        grpcserver
└── tasks.create
    └── INSERT


POST /mcp                               mcpserver (HTTP transport)
└── tools/call create_task              mcpserver
    └── tasks.create
        └── INSERT
```

## Metrics

| Metric                                                | From                                          |
| ----------------------------------------------------- | --------------------------------------------- |
| `http.server.request.duration`                        | [httpserver](/oryn/httpserver/observability/) |
| `rpc.server.call.duration`                            | [grpcserver](/oryn/grpcserver/observability/) |
| `mcp.server.operation.duration`                       | [mcpserver](/oryn/mcpserver/observability/)   |
| `db.client.operation.duration`, `db.sql.connection.*` | [sql](/oryn/sql/observability/)               |
| `tasks.created`, `tasks.completed`                    | `tasks.Service`                               |

## Logs

One log per request (`http request`, `grpc request`, `mcp request`), and one per change (`task created`, `task completed`, `task deleted`), all with the trace and span IDs of the request.

## See them

The default configuration only prints the logs. Choose another environment with `APP_ENV`:

| Command                              | Telemetry                                                    |
| ------------------------------------ | ------------------------------------------------------------ |
| `go run ./cmd/serve`                 | Readable logs on the console                                 |
| `APP_ENV=stdout go run ./cmd/serve`  | Traces, metrics and logs printed on stdout, as JSON          |
| `APP_ENV=compose go run ./cmd/serve` | OTLP to the Grafana stack of `compose.yaml`, with PostgreSQL |

To explore them in Grafana, start [PostgreSQL and grafana/otel-lgtm](https://github.com/grafana/docker-otel-lgtm) (an OpenTelemetry collector, Tempo, Prometheus and Loki):

```shell
docker compose up -d
APP_ENV=compose go run ./cmd/migrate
APP_ENV=compose go run ./cmd/serve
```

Open Grafana on [localhost:3000](http://localhost:3000): the traces are in Tempo, the metrics in Prometheus, the logs in Loki. The `X-Trace-Id` header of an HTTP response gives the trace to look for.

The standard `OTEL_*` variables work too, for example `OTEL_EXPORTER_OTLP_ENDPOINT` to send the telemetry elsewhere. See [OpenTelemetry](/oryn/core/opentelemetry/).

# Testing

> How the demo tests the tasks service and its HTTP, gRPC and MCP APIs in memory, asserting responses and telemetry.

The demo tests the whole application in memory: a SQLite database per test, API clients connected without network, and the telemetry exported in memory.

```shell
go test ./...
```

## The demotest helper

`demotest.Start` starts the application with [coretest](/oryn/core/testing/), and connects a client to each API:

internal/demotest/demotest.go

```go
func Start(t *testing.T, options ...fx.Option) *App {
  app := &App{t: t}


  var (
    grpcServer *grpc.Server
    mcpServer  *mcp.Server
  )


  coretest.Start(t, demo.App,
    append(options,
      config.AsOptions(config.WithValues(map[string]any{
        "sql.driver": "sqlite",
        "sql.dsn":    filepath.Join(t.TempDir(), "test.db"),
      })),
      sql.Migrate(),
      fx.Populate(&app.HTTPServer, &grpcServer, &mcpServer, &app.Spans, &app.Logs, &app.Metrics),
    )...,
  )


  app.GRPC = tasksv1.NewTasksServiceClient(grpcservertest.Dial(t, grpcServer)) // in memory, with bufconn
  app.MCP = mcpservertest.Connect(t, mcpServer)                                // in-memory MCP transport


  return app
}
```

## The service

```go
func TestService(t *testing.T) {
  t.Parallel()


  var service *tasks.Service


  app := demotest.Start(t, fx.Populate(&service))


  task, err := service.Create(t.Context(), "test_title")
  require.NoError(t, err)


  create := spantest.Find(app.Spans, spantest.WithName("tasks.create"), spantest.WithAttributeValue(tasks.TaskIDKey.String(task.ID)))
  require.Len(t, create, 1)
  spantest.HasSpan(t, app.Spans, spantest.WithName("INSERT"), spantest.WithParent(create[0].SpanContext.SpanID()))
  logtest.HasRecord(t, app.Logs, logtest.WithBody("task created"), logtest.WithAttributeValue("task_id", task.ID))
  assert.Equal(t, int64(1), metrictest.Int64Sum(t, app.Metrics, "tasks.created"))
}
```

## The APIs

Each API test calls the API, then checks that the `tasks.create` span is a child of the server span.

internal/http/handlers_test.go

```go
response := app.HTTP(http.MethodPost, "/api/tasks", `{"title": "test_title"}`)
require.Equal(t, http.StatusCreated, response.Code)


server := spantest.Find(app.Spans, spantest.WithName("POST /api/tasks"))
spantest.HasSpan(t, app.Spans, spantest.WithName("tasks.create"), spantest.WithParent(server[0].SpanContext.SpanID()))
```

internal/grpc/service_test.go

```go
task, err := app.GRPC.CreateTask(t.Context(), &tasksv1.CreateTaskRequest{Title: "test_title"})
require.NoError(t, err)


_, err = app.GRPC.GetTask(t.Context(), &tasksv1.GetTaskRequest{Id: "missing"})
assert.Equal(t, codes.NotFound, status.Code(err))


server := spantest.Find(app.Spans, spantest.WithName("tasks.v1.TasksService/CreateTask"))
```

internal/mcp/tools_test.go

```go
result, err := app.MCP.CallTool(t.Context(), &mcp.CallToolParams{
  Name:      "create_task",
  Arguments: mcpapi.CreateInput{Title: "test_title"},
})
require.NoError(t, err)
assert.False(t, result.IsError)


server := spantest.Find(app.Spans, spantest.WithName("tools/call create_task"))
```

## Replace dependencies with fx.Replace

The `tasks.Service` depends on a `tasks.Store` interface, provided by the SQL `Repository`:

internal/tasks/module.go

```go
var Module = fx.Module("tasks",
  fx.Provide(
    fx.Annotate(NewRepository, fx.As(new(Store))),
    NewService,
  ),
)
```

Tests replace it with [fx.Replace](https://pkg.go.dev/go.uber.org/fx#Replace), here with a store failing every call, to check how each API handles an unexpected error. The rest of the application is unchanged:

internal/demotest/demotest.go

```go
func ReplaceStore(store tasks.Store) fx.Option {
  return fx.Replace(fx.Annotate(store, fx.As(new(tasks.Store))))
}
```

internal/http/handlers_test.go

```go
app := demotest.Start(t, demotest.ReplaceStore(demotest.FailingStore{}))


response := app.HTTP(http.MethodGet, "/api/tasks", "")
assert.Equal(t, http.StatusInternalServerError, response.Code)
assert.JSONEq(t, `{"error": "internal error"}`, response.Body.String(), "the error is not exposed")


spantest.HasSpan(t, app.Spans, spantest.WithName("GET /api/tasks"), spantest.WithStatus(codes.Error))
logtest.HasRecord(t, app.Logs, logtest.WithBody("tasks request failed"), logtest.WithLevel(slog.LevelError))
```

The same failure is tested on every surface:

| Surface         | Result                                                                            |
| --------------- | --------------------------------------------------------------------------------- |
| `tasks.Service` | the error, a `tasks.create` span with an `Error` status, no `tasks.created` count |
| HTTP            | `500`, without the error details, and an error log                                |
| gRPC            | `INTERNAL`, a failed server span, and an error log                                |
| MCP             | a tool error, with `error.type=tool_error`, and an error log                      |

Replace any provided dependency the same way: a clock, an HTTP client, a fake of a third party API.

The requests go through the whole servers, with their instrumentation, middlewares and interceptors: the same code as in production, without ports. See the testing pages of [httpserver](/oryn/httpserver/testing/), [grpcserver](/oryn/grpcserver/testing/) and [mcpserver](/oryn/mcpserver/testing/).

# Getting started

> Create a first oryn application, with configuration, an HTTP endpoint, telemetry and a test.

This guide builds a small HTTP service: it answers `GET /hello/{name}`, and produces traces, metrics and logs.

Go 1.26 or later is required.

## Install

```shell
go mod init example.com/hello
go get github.com/go-oryn/oryn/core github.com/go-oryn/oryn/httpserver
```

## Configuration

Configuration files live in a `configs` directory, embedded in the binary:

```plaintext
hello/
├── configs/
│   ├── app.yaml
│   ├── embed.go
│   └── dev/
│       └── otel.yaml
├── main.go
└── main_test.go
```

configs/app.yaml

```yaml
app:
  name: hello
  version: 0.1.0
log:
  console:
    enabled: true
```

```yaml
# configs/dev/otel.yaml: loaded when APP_ENV=dev, prints the spans on the console
otel:
  traces:
    exporter: console
  metrics:
    exporter: none
  logs:
    exporter: none
```

configs/embed.go

```go
package configs


import "embed"


//go:embed *.yaml */*.yaml
var FS embed.FS
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

## Application

main.go

```go
package main


import (
  "fmt"
  "net/http"


  "github.com/go-oryn/oryn/core"
  "github.com/go-oryn/oryn/httpserver"


  "example.com/hello/configs"
)


// Module holds the application handlers. Tests reuse it.
var Module = httpserver.AsHandler("GET /hello/{name}", func(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, "hello %s", r.PathValue("name"))
})


func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    httpserver.Module,
    Module,
  ).Run(httpserver.Serve())
}
```

* `core.New` creates the application, with the core module: configuration, OpenTelemetry, logging and health checks.
* `httpserver.Module` provides the HTTP server, and `httpserver.AsHandler` registers a handler on it.
* `Run` starts the application with the HTTP server, and stops it gracefully on `SIGINT` or `SIGTERM`.

Run it:

```shell
APP_ENV=dev go run .
curl localhost:8080/hello/oryn
```

The console shows the request log, and the `GET /hello/{name}` span. Without `APP_ENV=dev`, telemetry is sent with OTLP to `localhost:4318`, or wherever `OTEL_EXPORTER_OTLP_ENDPOINT` points.

## Test

Tests run the application in the `test` environment, where telemetry is exported in memory, and serve requests without a network:

main_test.go

```go
package main


import (
  "net/http"
  "net/http/httptest"
  "testing"


  "github.com/stretchr/testify/assert"
  "go.opentelemetry.io/otel/sdk/trace/tracetest"
  "go.uber.org/fx"


  "github.com/go-oryn/oryn/core"
  "github.com/go-oryn/oryn/core/coretest"
  "github.com/go-oryn/oryn/core/otel/oteltest/spantest"
  "github.com/go-oryn/oryn/httpserver"


  "example.com/hello/configs"
)


func TestHello(t *testing.T) {
  t.Parallel()


  var (
    server *http.Server
    spans  *tracetest.InMemoryExporter
  )


  app := core.New(core.WithConfigFS(configs.FS), httpserver.Module, Module)
  coretest.Start(t, app, fx.Populate(&server, &spans))


  recorder := httptest.NewRecorder()
  server.Handler.ServeHTTP(recorder, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/hello/oryn", nil))


  assert.Equal(t, "hello oryn", recorder.Body.String())
  spantest.HasSpan(t, spans, spantest.WithName("GET /hello/{name}"))
}
```

```shell
go test ./...
```

## Next steps

* [Demo application](/oryn/demo/overview/): a complete application, serving one tasks module over HTTP, gRPC and MCP, with a database and its telemetry.
* [Application](/oryn/core/application/): how an oryn application is assembled.
* [HTTP server](/oryn/httpserver/overview/), [gRPC server](/oryn/grpcserver/overview/), [MCP server](/oryn/mcpserver/overview/): the servers of your APIs.
* [OpenTelemetry](/oryn/core/opentelemetry/): exporters, sampling and the `OTEL_*` variables.

# Configuration

> Every grpcclient configuration key, with its default and when to change it.

The `grpcclient.*` keys configure the gRPC client connections. Only the `target` of the used connections is required.

## Reference

```yaml
grpcclient:
  connections:
    <name>:
      target: "users:50051"   # required, gRPC target
      insecure: false         # plaintext instead of TLS with the system root certificates
      authority: ""           # :authority header, and TLS server name, "" = from the target
      service_config: ""      # default service config, as JSON
  log:
    enabled: true             # log every call
  observability:
    exclude: []               # full methods excluded from spans, metrics and logs
```

| Key                                            | Type   | Default                | Environment variable                               |
| ---------------------------------------------- | ------ | ---------------------- | -------------------------------------------------- |
| `grpcclient.connections.<name>.target`         | string | required               | `APP_GRPCCLIENT_CONNECTIONS_<NAME>_TARGET`         |
| `grpcclient.connections.<name>.insecure`       | bool   | `false`                | `APP_GRPCCLIENT_CONNECTIONS_<NAME>_INSECURE`       |
| `grpcclient.connections.<name>.authority`      | string | `""` (from the target) | `APP_GRPCCLIENT_CONNECTIONS_<NAME>_AUTHORITY`      |
| `grpcclient.connections.<name>.service_config` | string | `""`                   | `APP_GRPCCLIENT_CONNECTIONS_<NAME>_SERVICE_CONFIG` |
| `grpcclient.log.enabled`                       | bool   | `true`                 | `APP_GRPCCLIENT_LOG_ENABLED`                       |
| `grpcclient.observability.exclude`             | list   | empty                  |                                                    |

## Target

The target uses the [gRPC name syntax](https://github.com/grpc/grpc/blob/master/doc/naming.md). Without a scheme, it is resolved with DNS:

```yaml
grpcclient:
  connections:
    users:
      target: users:50051                       # DNS
    orders:
      target: dns:///orders.internal:443        # DNS, explicit
    local:
      target: unix:///var/run/local.sock        # Unix socket
```

A missing target fails the creation of the connection, and so the application start when the connection is used by `AsClient`.

## Transport security

Connections use TLS with the system root certificates by default. Use plaintext for servers without TLS, such as services in the same cluster behind a mesh, or in development:

```yaml
# configs/dev/grpcclient.yaml (APP_ENV=dev)
grpcclient:
  connections:
    users:
      insecure: true
```

For client certificates or custom root certificates, set the credentials with [`AsDialOptions`](/oryn/grpcclient/overview/#dial-options).

## Authority

The `:authority` header, and the TLS server name, default to the target host. Override them when calling through a proxy, or when the certificate does not match the target:

```yaml
grpcclient:
  connections:
    users:
      target: 10.0.0.12:443
      authority: users.example.com
```

## Service config

The [service config](https://github.com/grpc/grpc/blob/master/doc/service_config.md) sets the load balancing, the retries and the timeouts of a connection, as JSON. It is used unless the name resolver provides one:

```yaml
grpcclient:
  connections:
    users:
      target: dns:///users.internal:50051
      service_config: |
        {
          "loadBalancingConfig": [{"round_robin": {}}],
          "methodConfig": [{
            "name": [{"service": "users.v1.Users"}],
            "timeout": "2s",
            "retryPolicy": {
              "maxAttempts": 3,
              "initialBackoff": "0.1s",
              "maxBackoff": "1s",
              "backoffMultiplier": 2,
              "retryableStatusCodes": ["UNAVAILABLE"]
            }
          }]
        }
```

An invalid service config fails the creation of the connection.

## Call logs

A log is written for every call, `grpc client request`, see [Observability](/oryn/grpcclient/observability/#logs). Disable it when the traces are enough:

```yaml
grpcclient:
  log:
    enabled: false
```

## Exclude methods

No method is excluded by default. Excluded methods get no span, no metric and no call log:

```yaml
grpcclient:
  observability:
    exclude:
      - /grpc.health.v1.Health/Check
      - /users.v1.Users/Ping
```

Methods are full gRPC method names: `/<package>.<Service>/<Method>`. The trace context of the caller is still propagated for excluded methods.

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the gRPC client calls, and how to exclude methods.

Every call goes through, from the outermost:

1. the [otelgrpc](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc) instrumentation,
2. the call log,
3. the registered [interceptors](/oryn/grpcclient/overview/#interceptors).

## Traces

Each call gets a client span:

* named after the full method: `users.v1.Users/GetUser`,
* with the [RPC semantic conventions](https://opentelemetry.io/docs/specs/semconv/rpc/grpc/) attributes: `rpc.system.name`, `rpc.method` and `rpc.response.status_code`,
* with an error status for server errors: `Unknown`, `DeadlineExceeded`, `Unimplemented`, `Internal`, `Unavailable` and `DataLoss`.

The span is a child of the span of the call context. The trace context is injected in the call metadata with the application propagator, so the server span, of a [grpcserver](/oryn/grpcserver/overview/) application for example, continues the trace.

## Metrics

| Metric                     | Description                                                                            |
| -------------------------- | -------------------------------------------------------------------------------------- |
| `rpc.client.call.duration` | Duration of the calls, with the `rpc.method` and `rpc.response.status_code` attributes |

## Logs

Each call is logged as `grpc client request`, at error level for server errors (see above) and info level otherwise, with:

| Attribute  | Example                                     |
| ---------- | ------------------------------------------- |
| `rpc`      | `/users.v1.Users/GetUser`                   |
| `service`  | `users.v1.Users`                            |
| `method`   | `GetUser`                                   |
| `target`   | `users:50051`, the connection target        |
| `code`     | `NotFound`                                  |
| `error`    | `user not found`, the error message, if any |
| `duration` | `1.2ms`                                     |

Streaming calls are logged once the stream is created, with the code of its creation. The log is correlated with the call span. Disable call logs with `grpcclient.log.enabled: false`.

## Exclude methods

To exclude methods from spans, metrics and call logs, list their full methods:

```yaml
grpcclient:
  observability:
    exclude:
      - /grpc.health.v1.Health/Check
      - /users.v1.Users/Ping
```

# gRPC client

> The grpcclient module, gRPC client connections configured by name, with generated clients and interceptors registered through fx.

The `grpcclient` module provides [gRPC](https://grpc.io/docs/languages/go/) client connections, declared by name in the configuration and instrumented with OpenTelemetry.

```shell
go get github.com/go-oryn/oryn/grpcclient
```

## Quick start

configs/grpcclient.yaml

```yaml
grpcclient:
  connections:
    users:
      target: users:50051
      insecure: true
```

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    grpcclient.Module,
    grpcclient.AsClient("users", pb.NewUsersClient),
    fx.Provide(NewService),
  ).Run()
}


type Service struct {
  users pb.UsersClient
}


func NewService(users pb.UsersClient) *Service {
  return &Service{users: users}
}
```

* `grpcclient.Module` provides the `*grpcclient.Factory`, which creates the connections declared under `grpcclient.connections.<name>`.
* `grpcclient.AsClient("users", pb.NewUsersClient)` provides a `pb.UsersClient` on the `users` connection.
* Every call is traced, measured and logged, and the trace context is propagated to the server, see [Observability](/oryn/grpcclient/observability/).

## Connections

A connection is declared by its name, with at least a `target`, in the [gRPC name syntax](https://github.com/grpc/grpc/blob/master/doc/naming.md): `users:50051`, `dns:///users.internal:443`… It uses TLS with the system root certificates, unless `insecure` is `true`. See [Configuration](/oryn/grpcclient/configuration/).

The `*grpcclient.Factory` creates each connection once, on first use, and shares it. Creating a connection does not connect: gRPC connects on the first call, and reconnects as needed. All the connections are closed when the application stops.

```go
conn, err := factory.Conn("users")
if err != nil {
  return err // grpcclient.connections.users.target is missing, or the connection settings are invalid
}


client := pb.NewUsersClient(conn)
```

## Clients

`AsClient` provides the client returned by a generated `pb.NewXxxClient` constructor, on a named connection. The application fails to start if the connection cannot be created.

As fx identifies values by type, a client type can be provided only once. To call the same service on two connections, build the clients from the factory, and name them:

```go
fx.Provide(
  fx.Annotate(func(f *grpcclient.Factory) (pb.UsersClient, error) {
    conn, err := f.Conn("users-eu")
    if err != nil {
      return nil, err
    }


    return pb.NewUsersClient(conn), nil
  }, fx.ResultTags(`name:"users-eu"`)),
)
```

## Interceptors

Interceptors run code around every call of every connection: authentication, metadata, retries… Register them with `AsUnaryInterceptor` and `AsStreamInterceptor`, as values or constructors whose parameters are injected by fx:

```go
func NewAuthInterceptor(tokens *TokenSource) grpc.UnaryClientInterceptor {
  return func(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
    ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+tokens.Token())


    return invoker(ctx, method, req, reply, cc, opts...)
  }
}
```

```go
grpcclient.AsUnaryInterceptor(NewAuthInterceptor)
grpcclient.AsStreamInterceptor(NewAuthStreamInterceptor)
```

Interceptors apply in registration order: the first registered interceptor is the outermost one. They run after the built-in call log.

## Dial options

`AsDialOptions` adds [dial options](https://pkg.go.dev/google.golang.org/grpc#DialOption) to one connection, for what the configuration does not cover: custom TLS credentials, keepalive, a custom dialer…

```go
grpcclient.AsDialOptions("users",
  grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),
  grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 30 * time.Second}),
)
```

They are applied after the configured settings, so they override them.

## Configuration

The `grpcclient.*` keys declare the connections, the call logs and the methods excluded from telemetry. See [Configuration](/oryn/grpcclient/configuration/).

# Testing

> Test gRPC clients against in memory servers, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/). The `grpcclienttest` package serves a `*grpc.Server` in memory, and points a named connection at it, without listening on a port and whatever the configured target.

## Fake server

`grpcclienttest.Serve` serves a test server on the connection:

```go
type fakeUsers struct {
  pb.UnimplementedUsersServer
}


func (fakeUsers) GetUser(_ context.Context, req *pb.GetUserRequest) (*pb.User, error) {
  return &pb.User{Id: req.GetId(), Name: "test_name"}, nil
}


func TestService(t *testing.T) {
  t.Parallel()


  server := grpc.NewServer()
  pb.RegisterUsersServer(server, fakeUsers{})


  var (
    service *app.Service
    spans   *tracetest.InMemoryExporter
    logs    *logtest.InMemoryExporter
  )


  coretest.Start(t, app.New(), grpcclienttest.Serve("users", server), fx.Populate(&service, &spans, &logs))


  name, err := service.UserName(t.Context(), "42")
  require.NoError(t, err)
  assert.Equal(t, "test_name", name)


  spantest.HasSpan(t, spans,
    spantest.WithName("users.v1.Users/GetUser"),
    spantest.WithKind(trace.SpanKindClient),
  )
  logtest.HasRecord(t, logs, logtest.WithBody("grpc client request"), logtest.WithAttributeValue("method", "GetUser"))
}
```

The connection is plaintext, its target is `passthrough:///users`. The server is started with the application, and stopped with it.

## Application server

`grpcclienttest.ServeApp` serves the `*grpc.Server` of the application, provided by the [grpcserver](/oryn/grpcserver/overview/) module, on the connection. Use it to test a client against the real services, or the trace propagation:

```go
coretest.Start(t,
  core.New(
    grpcserver.Module,
    grpcserver.AsService(&pb.Users_ServiceDesc, NewUsersService),
    grpcclient.Module,
    grpcclient.AsClient("users", pb.NewUsersClient),
  ),
  grpcclienttest.ServeApp("users"),
  fx.Populate(&client, &spans),
)


_, err := client.GetUser(t.Context(), &pb.GetUserRequest{Id: "42"})
require.NoError(t, err)


clientSpans := spantest.Find(spans, spantest.WithName("users.v1.Users/GetUser"), spantest.WithKind(trace.SpanKindClient))
require.Len(t, clientSpans, 1)
spantest.HasSpan(t, spans,
  spantest.WithKind(trace.SpanKindServer),
  spantest.WithParent(clientSpans[0].SpanContext.SpanID()),
)
```

# Configuration

> Every grpcserver configuration key, with its default and when to change it.

The `grpcserver.*` keys configure the gRPC server. All of them are optional.

## Reference

```yaml
grpcserver:
  address: ":50051"           # listen address
  max_recv_msg_size: 0        # max received message size in bytes, 0 = gRPC default (4 MiB)
  max_send_msg_size: 0        # max sent message size in bytes, 0 = gRPC default (no limit)
  health:
    enabled: true             # register the grpc.health.v1 Health service
  reflection:
    enabled: false            # register the server reflection
  log:
    enabled: true             # log every call
  observability:
    exclude:                  # full methods excluded from spans, metrics and logs
      - /grpc.health.v1.Health/Check
      - /grpc.health.v1.Health/Watch
      - /grpc.health.v1.Health/List
      - /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
      - /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo
```

| Key                                | Type   | Default                       | Environment variable                |
| ---------------------------------- | ------ | ----------------------------- | ----------------------------------- |
| `grpcserver.address`               | string | `:50051`                      | `APP_GRPCSERVER_ADDRESS`            |
| `grpcserver.max_recv_msg_size`     | int    | `0` (4 MiB)                   | `APP_GRPCSERVER_MAX_RECV_MSG_SIZE`  |
| `grpcserver.max_send_msg_size`     | int    | `0` (no limit)                | `APP_GRPCSERVER_MAX_SEND_MSG_SIZE`  |
| `grpcserver.health.enabled`        | bool   | `true`                        | `APP_GRPCSERVER_HEALTH_ENABLED`     |
| `grpcserver.reflection.enabled`    | bool   | `false`                       | `APP_GRPCSERVER_REFLECTION_ENABLED` |
| `grpcserver.log.enabled`           | bool   | `true`                        | `APP_GRPCSERVER_LOG_ENABLED`        |
| `grpcserver.observability.exclude` | list   | health and reflection methods |                                     |

## Address

```yaml
grpcserver:
  address: ":50051"           # all interfaces
  # address: "127.0.0.1:0"    # a random free port, for tests
```

Once `grpcserver.Serve()` has started the server, the provided `*grpcserver.Address` holds the real listening address.

## Message sizes

gRPC rejects received messages larger than 4 MiB with `RESOURCE_EXHAUSTED`. Raise the limit for services exchanging large messages:

```yaml
grpcserver:
  max_recv_msg_size: 16777216   # 16 MiB
```

## Health

The [gRPC health service](https://github.com/grpc/grpc/blob/master/doc/health-checking.md) is registered by default, and answers with the application [health checks](/oryn/core/health/): the `""` and `readiness` services run the readiness probes, `liveness` and `startup` the others. Kubernetes can use it directly:

```yaml
# Kubernetes container spec
readinessProbe:
  grpc:
    port: 50051
```

Disable it if the application registers its own `grpc.health.v1.Health` service:

```yaml
grpcserver:
  health:
    enabled: false
```

## Reflection

[Server reflection](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md) lets tools like [grpcurl](https://github.com/fullstorydev/grpcurl) list and call the services without the proto files. It exposes the API description: enable it in development, or for internal services.

```yaml
# configs/dev/grpcserver.yaml (APP_ENV=dev)
grpcserver:
  reflection:
    enabled: true
```

```shell
grpcurl -plaintext localhost:50051 list
```

## Call logs

A log is written for every call, `grpc request`, see [Observability](/oryn/grpcserver/observability/#logs). Disable it when the traces are enough:

```yaml
grpcserver:
  log:
    enabled: false
```

## Exclude methods

The health checks and the reflection are excluded from spans, metrics and call logs by default. The list replaces the default one, so keep its entries when adding yours:

```yaml
grpcserver:
  observability:
    exclude:
      - /grpc.health.v1.Health/Check
      - /grpc.health.v1.Health/Watch
      - /grpc.health.v1.Health/List
      - /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
      - /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo
      - /users.v1.Users/Ping
```

Methods are full gRPC method names: `/<package>.<Service>/<Method>`. An empty list keeps the default.

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the gRPC server, and how to exclude methods.

Every call goes through, from the outermost:

1. the [otelgrpc](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc) instrumentation,
2. the call log,
3. the panic recovery,
4. the registered [interceptors](/oryn/grpcserver/overview/#interceptors) and the service.

## Traces

Each call gets a server span:

* named after the full method: `users.v1.Users/GetUser`,
* with the [RPC semantic conventions](https://opentelemetry.io/docs/specs/semconv/rpc/grpc/) attributes: `rpc.system.name`, `rpc.method` and `rpc.response.status_code`,
* with an error status for server errors: `Unknown`, `DeadlineExceeded`, `Unimplemented`, `Internal`, `Unavailable` and `DataLoss`.

The incoming trace context is extracted from the call metadata with the application propagator, so the span continues the trace of the caller. Spans created in services with the call context are children of the call span.

## Metrics

| Metric                     | Description                                                                            |
| -------------------------- | -------------------------------------------------------------------------------------- |
| `rpc.server.call.duration` | Duration of the calls, with the `rpc.method` and `rpc.response.status_code` attributes |

## Logs

Each call is logged as `grpc request`, at error level for server errors (see above) and info level otherwise, with:

| Attribute  | Example                                     |
| ---------- | ------------------------------------------- |
| `rpc`      | `/users.v1.Users/GetUser`                   |
| `service`  | `users.v1.Users`                            |
| `method`   | `GetUser`                                   |
| `code`     | `NotFound`                                  |
| `error`    | `user not found`, the error message, if any |
| `duration` | `1.2ms`                                     |

The log is correlated with the call span. Disable call logs with `grpcserver.log.enabled: false`.

## Panics

A panic in a service or an interceptor is logged as `grpc handler panic` at error level, with the `rpc`, `panic` and `stack` attributes, and answered with an `Internal` error. The span and the call log record the `Internal` code.

## Exclude methods

The health checks and the server reflection are excluded from spans, metrics and call logs by default. To exclude other methods, list their full methods; the list replaces the default one:

```yaml
grpcserver:
  observability:
    exclude:
      - /grpc.health.v1.Health/Check
      - /grpc.health.v1.Health/Watch
      - /grpc.health.v1.Health/List
      - /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
      - /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo
      - /users.v1.Users/Ping
```

# gRPC server

> The grpcserver module, a gRPC server with services and interceptors registered through fx, and the gRPC health checking protocol.

The `grpcserver` module provides a [gRPC](https://grpc.io/docs/languages/go/) server, instrumented with OpenTelemetry, with the gRPC health checking protocol backed by the [application health checks](/oryn/core/health/).

```shell
go get github.com/go-oryn/oryn/grpcserver
```

## Quick start

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    grpcserver.Module,
    grpcserver.AsService(&pb.Users_ServiceDesc, NewUsersService),
  ).Run(grpcserver.Serve())
}
```

* `grpcserver.Module` provides the `*grpc.Server`, with the registered services and interceptors, and its `*grpcserver.Address`.
* `grpcserver.Serve()` starts it with the application, and stops it gracefully when the application stops: in-flight calls are completed, unless the stop timeout expires first. If the server fails while running, the application stops.
* Every call is traced, measured and logged, see [Observability](/oryn/grpcserver/observability/).

## Services

Register the implementation of a service with its generated service description, `XXX_ServiceDesc`. The implementation is either a value, or a constructor whose parameters are injected by fx:

```go
type UsersService struct {
  pb.UnimplementedUsersServer


  repository *Repository
}


func NewUsersService(repository *Repository) *UsersService {
  return &UsersService{repository: repository}
}


func (s *UsersService) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
  user, err := s.repository.Get(ctx, req.GetId())
  if errors.Is(err, ErrNotFound) {
    return nil, status.Error(codes.NotFound, "user not found")
  }
  ...
}
```

```go
grpcserver.AsService(&pb.Users_ServiceDesc, NewUsersService)
```

The application fails to start if the implementation does not implement the service, or if a service is registered twice.

## Interceptors

Interceptors run code around every call: authentication, metadata, validation… Register them with `AsUnaryInterceptor` and `AsStreamInterceptor`, as values or constructors:

```go
func NewAuthInterceptor(auth *Authenticator) grpc.UnaryServerInterceptor {
  return func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
    if err := auth.Check(ctx); err != nil {
      return nil, status.Error(codes.Unauthenticated, err.Error())
    }


    return handler(ctx, req)
  }
}
```

```go
grpcserver.AsUnaryInterceptor(NewAuthInterceptor)
grpcserver.AsStreamInterceptor(NewAuthStreamInterceptor)
```

Interceptors apply to every service, in registration order: the first registered interceptor is the outermost one. They run after the built-in ones: the call log, then the panic recovery. A panic in a service or an interceptor is logged with its stack, and answered with an `Internal` error.

## Health

The server implements the [gRPC health checking protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md), so load balancers and Kubernetes [gRPC probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) can check it. The checks run the probes of the [health module](/oryn/core/health/):

| Service name      | Check     |
| ----------------- | --------- |
| `""` (the server) | readiness |
| `liveness`        | liveness  |
| `readiness`       | readiness |
| `startup`         | startup   |

Other service names answer `NotFound`. `Watch` is not implemented. As the readiness check fails once the application stops, so does the gRPC readiness check.

```shell
grpcurl -plaintext -d '{"service": "liveness"}' localhost:50051 grpc.health.v1.Health/Check
```

Disable it with `grpcserver.health.enabled: false`, for example to register your own health service.

## Reflection

The [server reflection](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md) lets clients like [grpcurl](https://github.com/fullstorydev/grpcurl) list and call the services without their proto files. It is disabled by default:

```yaml
grpcserver:
  reflection:
    enabled: true
```

## Address

The `*grpcserver.Address` is `grpcserver.address`, then the listening address once the server is started, useful with port `0`:

```go
var address *grpcserver.Address


coretest.Start(t, app, config.AsOptions(config.WithValues(map[string]any{"grpcserver.address": "127.0.0.1:0"})),
  grpcserver.Serve(), fx.Populate(&address))


conn, err := grpc.NewClient(address.String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
```

## Configuration

The `grpcserver.*` keys set the address, the message sizes, the health service, the reflection, the call logs and the methods excluded from telemetry. See [Configuration](/oryn/grpcserver/configuration/).

# Testing

> Test gRPC services in memory, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/), and call the `*grpc.Server` with `grpcservertest.Dial`, which serves it on an in memory connection, without listening on a port:

```go
func TestGetUser(t *testing.T) {
  t.Parallel()


  var (
    server *grpc.Server
    spans  *tracetest.InMemoryExporter
    logs   *logtest.InMemoryExporter
  )


  coretest.Start(t, core.New(grpcserver.Module, users.Module), fx.Populate(&server, &spans, &logs))


  client := pb.NewUsersClient(grpcservertest.Dial(t, server))


  user, err := client.GetUser(t.Context(), &pb.GetUserRequest{Id: "42"})
  require.NoError(t, err)
  assert.Equal(t, "42", user.GetId())


  spantest.HasSpan(t, spans,
    spantest.WithName("users.v1.Users/GetUser"),
    spantest.WithAttributeValue(semconv.RPCResponseStatusCode("OK")),
  )
  logtest.HasRecord(t, logs, logtest.WithBody("grpc request"), logtest.WithAttributeValue("method", "GetUser"))
}
```

The call goes through the whole server: instrumentation, interceptors and service. The connection is closed and the server stopped when the test ends.

## Over the network

To test with a real connection, set `grpcserver.address` to `127.0.0.1:0` and add `grpcserver.Serve()`: once started, the `*grpcserver.Address` is the listening address.

```go
var address *grpcserver.Address


coretest.Start(t, app,
  config.AsOptions(config.WithValues(map[string]any{"grpcserver.address": "127.0.0.1:0"})),
  grpcserver.Serve(),
  fx.Populate(&address),
)


conn, err := grpc.NewClient(address.String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
```

# Configuration

> Every httpclient configuration key, with its default and when to change it.

The `httpclient.*` keys configure the clients of the factory. All of them are optional.

## Reference

```yaml
httpclient:
  timeout: 30s                    # max duration of a request, response body included, 0 = none
  transport:
    max_idle_conns: 100           # max idle connections, all hosts, 0 = unlimited
    max_idle_conns_per_host: 100  # max idle connections per host
    max_conns_per_host: 0         # max connections per host, 0 = unlimited
    idle_conn_timeout: 90s        # max duration of an idle connection, 0 = none
    tls_handshake_timeout: 10s    # max duration of the TLS handshake, 0 = none
    response_header_timeout: 0s   # max duration to receive the response headers, 0 = none
    expect_continue_timeout: 1s   # max duration to wait for a 100-continue, 0 = send the body at once
  log:
    enabled: true                 # log every request
  observability:
    exclude: []                   # URL regular expressions excluded from spans, metrics and logs
```

| Key                                            | Type     | Default         | Environment variable                               |
| ---------------------------------------------- | -------- | --------------- | -------------------------------------------------- |
| `httpclient.timeout`                           | duration | `30s`           | `APP_HTTPCLIENT_TIMEOUT`                           |
| `httpclient.transport.max_idle_conns`          | int      | `100`           | `APP_HTTPCLIENT_TRANSPORT_MAX_IDLE_CONNS`          |
| `httpclient.transport.max_idle_conns_per_host` | int      | `100`           | `APP_HTTPCLIENT_TRANSPORT_MAX_IDLE_CONNS_PER_HOST` |
| `httpclient.transport.max_conns_per_host`      | int      | `0` (unlimited) | `APP_HTTPCLIENT_TRANSPORT_MAX_CONNS_PER_HOST`      |
| `httpclient.transport.idle_conn_timeout`       | duration | `90s`           | `APP_HTTPCLIENT_TRANSPORT_IDLE_CONN_TIMEOUT`       |
| `httpclient.transport.tls_handshake_timeout`   | duration | `10s`           | `APP_HTTPCLIENT_TRANSPORT_TLS_HANDSHAKE_TIMEOUT`   |
| `httpclient.transport.response_header_timeout` | duration | `0s` (none)     | `APP_HTTPCLIENT_TRANSPORT_RESPONSE_HEADER_TIMEOUT` |
| `httpclient.transport.expect_continue_timeout` | duration | `1s`            | `APP_HTTPCLIENT_TRANSPORT_EXPECT_CONTINUE_TIMEOUT` |
| `httpclient.log.enabled`                       | bool     | `true`          | `APP_HTTPCLIENT_LOG_ENABLED`                       |
| `httpclient.observability.exclude`             | list     | `[]`            |                                                    |

Durations use the Go syntax: `500ms`, `30s`, `2m`. An explicit `0` is kept: `timeout: 0s` disables the timeout, it does not restore the default. The transport keeps the other settings of `http.DefaultTransport`, like the `HTTP_PROXY` and `HTTPS_PROXY` environment variables.

## Timeout

`httpclient.timeout` bounds the whole request: connection, redirects, and reading the response body. It applies to every client of the factory, unless built with `httpclient.WithTimeout`. For a per-request deadline, use the request context instead:

```go
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()


request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
```

Disable it for streaming responses or long downloads, and bound the waiting for the server with `response_header_timeout`:

```yaml
httpclient:
  timeout: 0s
  transport:
    response_header_timeout: 10s
```

## Connection pool

The clients of the factory share the transport, and so its connections. The default keeps up to 100 idle connections per host, instead of 2 for `http.DefaultTransport`, so an application calling the same API concurrently reuses its connections. To protect a fragile API, limit the connections to it:

```yaml
httpclient:
  transport:
    max_conns_per_host: 20
```

Requests over the limit wait for a free connection.

## Request logs

A log is written for every request, `http client request`, see [Observability](/oryn/httpclient/observability/#logs). Disable it when the traces are enough, or to reduce the volume:

```yaml
httpclient:
  log:
    enabled: false
```

## Exclude URLs

Exclude the calls to some URLs, like the polling of a status endpoint, with Go [regular expressions](https://pkg.go.dev/regexp/syntax), matched against the request URL without its query: `https://status.example.com/health`.

```yaml
httpclient:
  observability:
    exclude:
      - "^https://status\\.example\\.com/"
      - "/health$"
```

Excluded requests are still sent: they only get no span, no metrics and no request log. The trace context is not sent with them. An invalid expression fails the application start.

## Per environment

configs/app.yaml

```yaml
httpclient:
  timeout: 10s


# configs/dev/httpclient.yaml (APP_ENV=dev)
httpclient:
  timeout: 0s
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the HTTP clients, trace context propagation, and how to exclude URLs.

Every request goes through, from the outermost:

1. the registered [middlewares](/oryn/httpclient/overview/#middlewares),
2. the [otelhttp](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp) instrumentation,
3. the request log,
4. the base transport.

The instrumentation uses the tracer provider, meter provider and propagator of the application, never the OpenTelemetry globals.

## Traces

Each request gets a client span:

* named after the method, `GET`, or `HTTP` for a non-standard method, following the [HTTP semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client),
* with the `http.request.method`, `url.full`, `server.address`, `server.port` and `http.response.status_code` attributes; `url.full` keeps the query, without the user info and the values of known signature parameters,
* with an error status for transport errors and `5xx` responses.

The span is a child of the span of the request context: build the requests with `http.NewRequestWithContext`. The span ends when the response body is closed or fully read.

## Propagation

The trace context is injected in the request headers with the application propagator, `traceparent` and `tracestate` by default. A server instrumented with OpenTelemetry, like the [httpserver](/oryn/httpserver/overview/) module, continues the trace: its server span is a child of the client span.

## Metrics

| Metric                          | Description                                                                                                                          |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `http.client.request.duration`  | Duration of the requests, with the `http.request.method`, `server.address`, `server.port` and `http.response.status_code` attributes |
| `http.client.request.body.size` | Size of the request bodies, with the same attributes                                                                                 |

## Logs

Each request is logged as `http client request`, at error level for transport errors and `5xx` responses, and info level otherwise, with:

| Attribute  | Example                                                                               |
| ---------- | ------------------------------------------------------------------------------------- |
| `method`   | `GET`                                                                                 |
| `url`      | `https://weather.example.com/cities/paris`, without the user info, query and fragment |
| `status`   | `200`, unless the request failed                                                      |
| `error`    | `dial tcp: connection refused`, if the request failed                                 |
| `duration` | `1.2ms`, until the response headers are received                                      |

The query is left out of the log as it often holds secrets, like API keys: avoid secrets in URLs, the span `url.full` attribute keeps it. The log is correlated with the client span. Disable request logs with `httpclient.log.enabled: false`.

## Exclude URLs

To exclude requests from spans, metrics and logs, list URL regular expressions:

```yaml
httpclient:
  observability:
    exclude:
      - "^https://status\\.example\\.com/"
```

See [Configuration](/oryn/httpclient/configuration/#exclude-urls).

# HTTP client

> The httpclient module, HTTP clients instrumented with OpenTelemetry, with transport middlewares registered through fx.

The `httpclient` module provides standard `*http.Client`s, instrumented with OpenTelemetry: every request gets a client span, metrics and a log, and carries the trace context to the server it calls.

```shell
go get github.com/go-oryn/oryn/httpclient
```

## Quick start

```go
type WeatherClient struct {
  client *http.Client
}


func NewWeatherClient(client *http.Client) *WeatherClient {
  return &WeatherClient{client: client}
}


func (c *WeatherClient) Temperature(ctx context.Context, city string) (*Temperature, error) {
  request, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://weather.example.com/cities/"+url.PathEscape(city), nil)
  if err != nil {
    return nil, err
  }


  response, err := c.client.Do(request)
  ...
}
```

```go
core.New(
  httpclient.Module,
  fx.Provide(NewWeatherClient),
)
```

* `httpclient.Module` provides the default `*http.Client`, and the `*httpclient.Factory` to build more clients.
* Build the requests with the context of the caller, `http.NewRequestWithContext`: the client span is then a child of the current span, and the request is canceled with it.
* Every request is traced, measured and logged, see [Observability](/oryn/httpclient/observability/).
* The client has a 30 seconds timeout by default, see [Configuration](/oryn/httpclient/configuration/).

## Factory

The `*httpclient.Factory` builds clients with their own settings. They share the registered middlewares and the instrumentation:

```go
func NewWeatherClient(factory *httpclient.Factory) *WeatherClient {
  return &WeatherClient{client: factory.NewClient(
    httpclient.WithTimeout(5*time.Second),
    httpclient.WithMiddlewares(NewAPIKeyMiddleware("weather")),
  )}
}
```

| Option                  | Description                                                                                                                                    |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `WithTimeout(d)`        | The client timeout, instead of `httpclient.timeout`. `0` means none.                                                                           |
| `WithMiddlewares(m...)` | Middlewares of this client only, applied after the registered ones.                                                                            |
| `WithTransport(rt)`     | The base transport, instead of the configured one: a custom TLS configuration, a proxy… The middlewares and the instrumentation still wrap it. |

## Middlewares

Middlewares run code around every request: authentication headers, retries, rate limiting… A middleware implements `httpclient.Middleware`, or is a `func(http.RoundTripper) http.RoundTripper`. `httpclient.RoundTripperFunc` keeps them short:

```go
func NewAuthMiddleware(tokens *TokenSource) httpclient.Middleware {
  return httpclient.MiddlewareFunc(func(next http.RoundTripper) http.RoundTripper {
    return httpclient.RoundTripperFunc(func(r *http.Request) (*http.Response, error) {
      token, err := tokens.Token(r.Context())
      if err != nil {
        return nil, err
      }


      // A transport must not modify the request it receives: clone it.
      r = r.Clone(r.Context())
      r.Header.Set("Authorization", "Bearer "+token)


      return next.RoundTrip(r)
    })
  })
}
```

Register them with `AsMiddleware`, as values or constructors whose parameters are injected by fx:

```go
httpclient.AsMiddleware(NewAuthMiddleware)
```

Registered middlewares apply to every client of the factory, in registration order: the first registered middleware is the outermost one. A request goes through, from the outermost:

1. the registered middlewares, then the `WithMiddlewares` ones,
2. the OpenTelemetry instrumentation,
3. the request log,
4. the base transport.

As the middlewares wrap the instrumentation, a middleware retrying a request produces one client span per attempt.

## Configuration

The `httpclient.*` keys set the timeout, the connection pool, the request logs and the URLs excluded from telemetry. See [Configuration](/oryn/httpclient/configuration/).

# Testing

> Serve the requests of the HTTP clients in memory, and assert their telemetry.

The `httpclienttest` package routes the requests of the clients to an `http.Handler`, in memory: tests need no network nor fake server, and the requests still go through the middlewares, the instrumentation and the request log.

## Serve

`httpclienttest.Serve` replaces the base transport of the application: every client of the factory sends its requests to the handler.

```go
func TestTemperature(t *testing.T) {
  t.Parallel()


  var (
    weather *WeatherClient
    spans   *tracetest.InMemoryExporter
    logs    *logtest.InMemoryExporter
  )


  coretest.Start(t, core.New(httpclient.Module, fx.Provide(NewWeatherClient)),
    httpclienttest.Serve(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      assert.Equal(t, "/cities/paris", r.URL.Path)


      fmt.Fprint(w, `{"celsius": 21}`)
    })),
    fx.Populate(&weather, &spans, &logs),
  )


  temperature, err := weather.Temperature(t.Context(), "paris")
  require.NoError(t, err)
  assert.Equal(t, 21, temperature.Celsius)


  spantest.HasSpan(t, spans,
    spantest.WithName("GET"),
    spantest.WithAttributeValue(semconv.HTTPResponseStatusCode(http.StatusOK)),
  )
  logtest.HasRecord(t, logs, logtest.WithBody("http client request"), logtest.WithAttributeValue("status", "200"))
}
```

The handler gets a server request, like from a real server: `r.Host`, `r.RequestURI` and `r.RemoteAddr` (`httpclienttest.RemoteAddr`) are set. Route the requests of several APIs with an `http.ServeMux`, using host patterns: `mux.Handle("weather.example.com/", ...)`.

A client built with `httpclient.WithTransport` keeps its own transport: `Serve` does not apply to it.

## Transport

`httpclienttest.Transport` returns the in-memory `http.RoundTripper`, for a single client:

```go
client := factory.NewClient(httpclient.WithTransport(httpclienttest.Transport(handler)))
```

## Calling the application server

To test a client against the HTTP server of the application, serve its requests with the server handler. The trace context is propagated, so the server span is a child of the client span:

```go
var (
  factory *httpclient.Factory
  server  *http.Server
  spans   *tracetest.InMemoryExporter
)


coretest.Start(t, core.New(httpclient.Module, httpserver.Module, users.Module), fx.Populate(&factory, &server, &spans))


client := factory.NewClient(httpclient.WithTransport(httpclienttest.Transport(server.Handler)))
```

## Errors

Test the transport errors with a failing transport:

```go
client := factory.NewClient(httpclient.WithTransport(httpclient.RoundTripperFunc(func(*http.Request) (*http.Response, error) {
  return nil, errors.New("connection refused")
})))
```

A canceled request context fails the request with the context error, like with a real transport.

# Configuration

> Every httpserver configuration key, with its default and when to change it.

The `httpserver.*` keys configure the HTTP server. All of them are optional.

## Reference

```yaml
httpserver:
  address: ":8080"            # listen address
  read_header_timeout: 10s    # max duration to read the request headers, 0 = read_timeout
  read_timeout: 0s            # max duration to read the whole request, 0 = none
  write_timeout: 0s           # max duration to write the response, 0 = none
  idle_timeout: 2m            # max duration of an idle keep-alive connection, 0 = read_timeout
  log:
    enabled: true             # log every request
  observability:
    exclude: []               # path regular expressions excluded from spans, metrics and logs
```

| Key                                | Type     | Default     | Environment variable                 |
| ---------------------------------- | -------- | ----------- | ------------------------------------ |
| `httpserver.address`               | string   | `:8080`     | `APP_HTTPSERVER_ADDRESS`             |
| `httpserver.read_header_timeout`   | duration | `10s`       | `APP_HTTPSERVER_READ_HEADER_TIMEOUT` |
| `httpserver.read_timeout`          | duration | `0s` (none) | `APP_HTTPSERVER_READ_TIMEOUT`        |
| `httpserver.write_timeout`         | duration | `0s` (none) | `APP_HTTPSERVER_WRITE_TIMEOUT`       |
| `httpserver.idle_timeout`          | duration | `2m`        | `APP_HTTPSERVER_IDLE_TIMEOUT`        |
| `httpserver.log.enabled`           | bool     | `true`      | `APP_HTTPSERVER_LOG_ENABLED`         |
| `httpserver.observability.exclude` | list     | `[]`        |                                      |

Durations use the Go syntax: `500ms`, `30s`, `2m`. `0s` disables a timeout: like in `net/http`, a disabled `read_header_timeout` or `idle_timeout` falls back to `read_timeout`.

## Address

```yaml
httpserver:
  address: ":8080"          # all interfaces
  # address: "127.0.0.1:8080"  # local only
  # address: "127.0.0.1:0"     # a random free port, for tests
```

Once `httpserver.Serve()` has started the server, `server.Addr` holds the real listening address, which is useful with port `0`.

## Timeouts

The defaults protect against slow clients sending headers (`read_header_timeout`), and close idle keep-alive connections (`idle_timeout`). `read_timeout` and `write_timeout` are disabled, so long uploads, downloads and streaming responses work.

For an API with small requests and responses, set them to bound every request:

```yaml
httpserver:
  read_timeout: 30s
  write_timeout: 30s
```

`write_timeout` includes the handler time: a handler slower than it gets its connection closed.

## Request logs

A log is written for every request, `http request`, see [Observability](/oryn/httpserver/observability/#logs). Disable it when the traces are enough, or to reduce the volume:

```yaml
httpserver:
  log:
    enabled: false
```

## Exclude paths

Health checks and metric scrapes produce a lot of uninteresting telemetry. Exclude their paths with Go [regular expressions](https://pkg.go.dev/regexp/syntax), matched against the request path:

```yaml
httpserver:
  observability:
    exclude:
      - "^/livez$"
      - "^/readyz$"
      - "^/static/"
```

Excluded requests are still served: they only get no span, no metrics and no request log. An invalid expression fails the application start.

## Per environment

configs/app.yaml

```yaml
httpserver:
  address: ":8080"


# configs/dev/httpserver.yaml (APP_ENV=dev)
httpserver:
  address: "127.0.0.1:8080"
  log:
    enabled: true
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the HTTP server, and how to exclude paths.

Every request goes through, from the outermost:

1. the [otelhttp](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp) instrumentation,
2. the request log,
3. the panic recovery,
4. the routing, the [middlewares](/oryn/httpserver/routing/#middlewares) and the handler.

## Traces

Each request gets a server span:

* named after the route: `GET /users/{id}`, or just the method when no route matches,
* with the [HTTP semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/) attributes: `http.route`, `http.request.method`, `http.response.status_code`, `url.path`, `client.address`, body sizes…
* with an error status for `5xx` responses.

The incoming trace context (`traceparent`, `baggage`…) is extracted with the application propagator, so the span continues the trace of the caller. Spans created in handlers with `r.Context()` are children of the request span.

## Metrics

The HTTP server metrics of the semantic conventions, with the `http.route`, `http.request.method` and `http.response.status_code` attributes:

| Metric                           | Description                 |
| -------------------------------- | --------------------------- |
| `http.server.request.duration`   | Duration of the requests    |
| `http.server.request.body.size`  | Size of the request bodies  |
| `http.server.response.body.size` | Size of the response bodies |

## Logs

Each request is logged as `http request`, at error level for `5xx` responses and info level otherwise, with:

| Attribute       | Example           |
| --------------- | ----------------- |
| `method`        | `GET`             |
| `path`          | `/users/42`       |
| `route`         | `GET /users/{id}` |
| `status`        | `200`             |
| `response_size` | `512`             |
| `duration`      | `1.2ms`           |

The log is correlated with the request span. Disable request logs with `httpserver.log.enabled: false`.

## Panics

A panic in a handler or a middleware is logged at error level, with its stack, and answered with a `500` if nothing was written yet. The span and the request log record the `500`. A `panic(http.ErrAbortHandler)` aborts the response, as with the standard server.

## Exclude paths

Requests to health checks or other noisy endpoints can be excluded from spans, metrics and request logs, with Go regular expressions matched against the request path:

```yaml
httpserver:
  observability:
    exclude:
      - "^/livez$"
      - "^/readyz$"
```

# HTTP server

> The httpserver module, an HTTP server on net/http with handlers, middlewares and handler groups registered through fx.

The `httpserver` module provides an HTTP server built on the standard [http.ServeMux](https://pkg.go.dev/net/http#ServeMux), instrumented with OpenTelemetry.

```shell
go get github.com/go-oryn/oryn/httpserver
```

## Quick start

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    httpserver.Module,
    httpserver.AsHandler("GET /hello/{name}", func(w http.ResponseWriter, r *http.Request) {
      fmt.Fprintf(w, "hello %s", r.PathValue("name"))
    }),
  ).Run(httpserver.Serve())
}
```

* `httpserver.Module` provides the `*http.Server`, with the registered handlers and middlewares.
* `httpserver.Serve()` starts it with the application, and shuts it down gracefully when the application stops: in-flight requests are completed. If the server fails while running, the application stops.
* Handlers, middlewares and handler groups are registered with `AsHandler`, `AsMiddleware` and `AsHandlerGroup`, see [Routing](/oryn/httpserver/routing/).
* Every request is traced, measured and logged, see [Observability](/oryn/httpserver/observability/).

## Configuration

The `httpserver.*` keys set the address, the timeouts, the request logs and the paths excluded from telemetry. See [Configuration](/oryn/httpserver/configuration/).

# Routing

> Register handlers, middlewares and handler groups on the HTTP server.

## Handlers

`AsHandler` registers a handler for a [ServeMux pattern](https://pkg.go.dev/net/http#hdr-Patterns): `[METHOD ][HOST]/[PATH]`, with `{name}` wildcards read with `r.PathValue("name")`.

A handler is either:

* a value: an `http.Handler`, or a `func(http.ResponseWriter, *http.Request)`,
* or a constructor returning one, whose parameters are injected by fx.

```go
type GetUserHandler struct {
  repository *UserRepository
}


func NewGetUserHandler(repository *UserRepository) *GetUserHandler {
  return &GetUserHandler{repository: repository}
}


func (h *GetUserHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  user, err := h.repository.Find(r.Context(), r.PathValue("id"))
  if err != nil {
    http.Error(w, "user not found", http.StatusNotFound)


    return
  }


  _ = json.NewEncoder(w).Encode(user)
}


var Module = fx.Options(
  fx.Provide(NewUserRepository),
  httpserver.AsHandler("GET /users/{id}", NewGetUserHandler),
  httpserver.AsHandler("GET /ping", func(w http.ResponseWriter, _ *http.Request) {
    _, _ = io.WriteString(w, "pong")
  }),
)
```

Useful patterns:

| Pattern            | Matches                                           |
| ------------------ | ------------------------------------------------- |
| `GET /users/{id}`  | `GET` (and `HEAD`) `/users/42`                    |
| `POST /users`      | `POST /users` only                                |
| `/files/{path...}` | Any method, `/files/a/b/c`, with `path` = `a/b/c` |
| `GET /{$}`         | `GET /` only, not every path                      |
| `GET /static/`     | `GET` on every path under `/static/`              |

The application fails to start if a pattern is invalid, or conflicts with another one.

## Middlewares

A middleware is either:

* a value: a `func(http.Handler) http.Handler` (the usual Go middleware signature, so existing middlewares work as is), or an `httpserver.Middleware` (an interface with `Wrap(next http.Handler) http.Handler`),
* or a constructor returning one, whose parameters are injected by fx.

`AsMiddleware` registers global middlewares, applied to every handler. Middlewares can also be given to a single handler, after its pattern and handler:

```go
func NewAuthMiddleware(verifier *TokenVerifier) func(http.Handler) http.Handler {
  return func(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      if !verifier.Verify(r) {
        http.Error(w, "unauthorized", http.StatusUnauthorized)


        return
      }


      next.ServeHTTP(w, r)
    })
  }
}


var Module = fx.Options(
  httpserver.AsMiddleware(NewAuthMiddleware),
  httpserver.AsHandler("DELETE /users/{id}", NewDeleteUserHandler, NewAuditMiddleware),
)
```

Middlewares run in registration order, the first one being the outermost:

1. global middlewares,
2. group middlewares, outer groups first,
3. handler middlewares.

They run once the request is routed: `r.Pattern` and `r.PathValue()` are available. Requests matching no route (`404`, `405`) do not go through them.

## Handler groups

`AsHandlerGroup` registers handlers sharing a path prefix and middlewares. Groups can be nested with `WithGroup`.

```go
httpserver.AsHandlerGroup("/api/v1",
  httpserver.WithMiddlewares(NewAuthMiddleware),
  httpserver.WithHandler("GET /users", NewListUsersHandler),
  httpserver.WithHandler("POST /users", NewCreateUserHandler, NewAuditMiddleware),
  httpserver.WithGroup("/admin",
    httpserver.WithMiddlewares(NewAdminMiddleware),
    httpserver.WithHandler("GET /stats", NewStatsHandler),
  ),
)
```

| Request                   | Middlewares, in order |
| ------------------------- | --------------------- |
| `GET /api/v1/users`       | global, auth          |
| `POST /api/v1/users`      | global, auth, audit   |
| `GET /api/v1/admin/stats` | global, auth, admin   |

| Option                                          | Description                                                                                 |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `WithHandler(pattern, handler, middlewares...)` | Adds a handler, with its own middlewares                                                    |
| `WithMiddlewares(middlewares...)`               | Adds middlewares to all the handlers of the group and its sub groups, wherever it is placed |
| `WithGroup(prefix, options...)`                 | Adds a sub group, whose prefix is relative to the group one                                 |

* The group patterns are relative to the prefix: `GET /users` in the group `/api/v1` serves `GET /api/v1/users`. They cannot have a host.
* A group middleware is built once, and shared by the group handlers.

# Testing

> Test HTTP handlers in memory, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/), and serve requests with the handler of the `*http.Server`, without listening on a port:

```go
func TestGetUser(t *testing.T) {
  t.Parallel()


  var (
    server *http.Server
    spans  *tracetest.InMemoryExporter
    logs   *logtest.InMemoryExporter
  )


  coretest.Start(t, core.New(httpserver.Module, users.Module), fx.Populate(&server, &spans, &logs))


  recorder := httptest.NewRecorder()
  server.Handler.ServeHTTP(recorder, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/users/42", nil))


  assert.Equal(t, http.StatusOK, recorder.Code)
  spantest.HasSpan(t, spans,
    spantest.WithName("GET /users/{id}"),
    spantest.WithAttributeValue(semconv.HTTPResponseStatusCode(http.StatusOK)),
  )
  logtest.HasRecord(t, logs, logtest.WithBody("http request"), logtest.WithAttributeValue("route", "GET /users/{id}"))
}
```

The request goes through the whole server: instrumentation, middlewares and handler.

## Over the network

To test with a real HTTP client, set `httpserver.address` to `127.0.0.1:0` and add `httpserver.Serve()`: once started, `server.Addr` is the listening address.

```go
coretest.Start(t, app,
  config.AsOptions(config.WithValues(map[string]any{"httpserver.address": "127.0.0.1:0"})),
  httpserver.Serve(),
  fx.Populate(&server),
)


response, err := http.Get("http://" + server.Addr + "/users/42")
```

# Configuration

> Every mcpclient configuration key, with its default and when to change it.

The `mcpclient.*` keys configure the MCP servers to connect to, and the client telemetry.

## Reference

```yaml
mcpclient:
  servers:
    <name>:                   # the server name, given to factory.Session
      url: ""                 # streamable HTTP endpoint, required
      headers: {}             # HTTP headers sent with every request
      timeout: 0s             # max duration of a request, 0 = no timeout
  log:
    enabled: true             # log every request
  observability:
    exclude: []               # MCP methods excluded from spans, metrics and logs
```

| Key                                | Type     | Default  | Environment variable                   |
| ---------------------------------- | -------- | -------- | -------------------------------------- |
| `mcpclient.servers.<name>.url`     | string   | required | `APP_MCPCLIENT_SERVERS_<NAME>_URL`     |
| `mcpclient.servers.<name>.headers` | map      | `{}`     |                                        |
| `mcpclient.servers.<name>.timeout` | duration | `0s`     | `APP_MCPCLIENT_SERVERS_<NAME>_TIMEOUT` |
| `mcpclient.log.enabled`            | bool     | `true`   | `APP_MCPCLIENT_LOG_ENABLED`            |
| `mcpclient.observability.exclude`  | list     | `[]`     |                                        |

The client name and version sent to the servers are [`app.name` and `app.version`](/oryn/core/configuration/#keys).

## Servers

Each entry of `mcpclient.servers` is a server, opened with `factory.Session(ctx, "<name>")`. Its `url` is the [streamable HTTP](https://modelcontextprotocol.io/specification/latest/basic/transports#streamable-http) endpoint of the server:

```yaml
mcpclient:
  servers:
    docs:
      url: https://docs.example.com/mcp
    tasks:
      url: http://tasks:8082/mcp
```

`Session` fails for a name without `url`, unless a [custom transport](/oryn/mcpclient/overview/#custom-transports) is registered for it, and for an invalid URL. Names are case-insensitive.

## Headers

Headers are sent with every HTTP request to the server, typically for authentication. Use `${VAR}` placeholders to keep secrets out of the files:

```yaml
mcpclient:
  servers:
    docs:
      url: https://docs.example.com/mcp
      headers:
        Authorization: Bearer ${DOCS_TOKEN}
        X-Tenant: ${TENANT:default}
```

Header names are case-insensitive, like in HTTP.

## Timeout

By default, a request lasts as long as its context. Set `timeout` to bound every request to the server, from the call to the response:

```yaml
mcpclient:
  servers:
    docs:
      url: https://docs.example.com/mcp
      timeout: 10s
```

A request over the timeout fails with `context.DeadlineExceeded`. The timeout applies to each request, not to the session, which stays open.

## Request logs

A log is written for every MCP request, `mcp client request`, see [Observability](/oryn/mcpclient/observability/#logs). Disable it when the traces are enough:

```yaml
mcpclient:
  log:
    enabled: false
```

## Exclude methods

Exclude MCP methods from spans, metrics and request logs, for example the keep-alive pings and the discovery calls:

```yaml
mcpclient:
  observability:
    exclude:
      - ping
      - tools/list
```

Methods are the MCP method names: `server/discover`, `tools/list`, `tools/call`, `resources/read`, `prompts/get`…

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the MCP client, following the MCP semantic conventions, and trace propagation to the servers.

Every MCP request goes through, from the outermost:

1. the timeout, if configured,
2. the OpenTelemetry instrumentation, following the [MCP semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/),
3. the request log,
4. the transport.

With the streamable HTTP transport, the HTTP requests are also instrumented with [otelhttp](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp): an HTTP client span per request, and the `http.client.*` metrics.

## Traces

Each MCP request gets a client span, child of the span of `ctx`:

* named `{mcp.method.name} {target}`: `tools/call search`, `resources/read docs://readme`, `prompts/get review`, or just the method without target: `tools/list`,
* with the attributes:

| Attribute               | Example                          | When             |
| ----------------------- | -------------------------------- | ---------------- |
| `mcp.method.name`       | `tools/call`                     | always           |
| `mcp.session.id`        | `4XZ…`                           | with a session   |
| `gen_ai.operation.name` | `execute_tool`                   | tool calls       |
| `gen_ai.tool.name`      | `search`                         | tool calls       |
| `gen_ai.prompt.name`    | `review`                         | `prompts/get`    |
| `mcp.resource.uri`      | `docs://readme`                  | `resources/read` |
| `server.address`        | `docs.example.com`               | streamable HTTP  |
| `server.port`           | `443`                            | streamable HTTP  |
| `error.type`            | `tool_error`, `-32602`, `_OTHER` | failures         |

* with an error status when the request fails: `error.type` is the JSON-RPC error code for protocol errors (unknown tool, invalid params…), `_OTHER` for other errors (timeout, connection…), and `tool_error` for a tool returning an error result.

## Propagation

The trace continues on the server, in two ways:

* the trace context (`traceparent`, `tracestate`) is put in the request `_meta`, as advised by the conventions: the MCP server span of an [mcpserver](/oryn/mcpserver/observability/) is a child of the client span, whatever the transport,
* with streamable HTTP, the HTTP requests carry the trace context headers too.

The application propagator is used, so `OTEL_PROPAGATORS` applies.

## Metrics

| Metric                          | Description                              | Attributes                                                                                                 |
| ------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `mcp.client.operation.duration` | Duration of the MCP requests, in seconds | `mcp.method.name`, `gen_ai.tool.name`, `gen_ai.prompt.name`, `error.type`, `server.address`, `server.port` |

The instrumentation scope of the spans and metrics is `github.com/go-oryn/oryn/mcpclient`.

## Logs

Each request is logged as `mcp client request`, at error level for failures and tool errors, and info level otherwise, with:

| Attribute    | Example                         |
| ------------ | ------------------------------- |
| `server`     | `docs`, the server name         |
| `method`     | `tools/call`                    |
| `target`     | `search`                        |
| `session_id` | `4XZ…`                          |
| `duration`   | `1.2ms`                         |
| `error_type` | `tool_error` (failures only)    |
| `error`      | `missing query` (failures only) |

The log is correlated with the request span. Disable request logs with `mcpclient.log.enabled: false`. The logs of the MCP SDK itself are logged at debug level.

## Exclude methods

Noisy methods can be excluded from spans, metrics and request logs:

```yaml
mcpclient:
  observability:
    exclude:
      - ping
      - notifications/initialized
```

# MCP client

> The mcpclient module, Model Context Protocol client sessions to the configured MCP servers, instrumented with OpenTelemetry.

The `mcpclient` module opens [Model Context Protocol](https://modelcontextprotocol.io) (MCP) client sessions to the MCP servers of the configuration, built on the official [MCP Go SDK](https://github.com/modelcontextprotocol/go-sdk), instrumented with OpenTelemetry. It lets your application call the tools, read the resources and get the prompts of other MCP servers.

```shell
go get github.com/go-oryn/oryn/mcpclient
```

## Quick start

Name the servers in the configuration:

```yaml
mcpclient:
  servers:
    docs:
      url: https://docs.example.com/mcp
      headers:
        Authorization: Bearer ${DOCS_TOKEN}
```

Add the module, and get the sessions from the `*mcpclient.Factory`:

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    mcpclient.Module,
    fx.Provide(NewSearch),
  ).Run()
}


type Search struct {
  factory *mcpclient.Factory
}


func NewSearch(factory *mcpclient.Factory) *Search {
  return &Search{factory: factory}
}


func (s *Search) Search(ctx context.Context, query string) (*mcp.CallToolResult, error) {
  session, err := s.factory.Session(ctx, "docs")
  if err != nil {
    return nil, err
  }


  return session.CallTool(ctx, &mcp.CallToolParams{
    Name:      "search",
    Arguments: map[string]any{"query": query},
  })
}
```

* `mcpclient.Module` provides the `*mcpclient.Factory`.
* Every request is traced, measured and logged, and the trace continues on the server, see [Observability](/oryn/mcpclient/observability/).
* Servers are reached over the [streamable HTTP transport](https://modelcontextprotocol.io/specification/latest/basic/transports#streamable-http), see [Configuration](/oryn/mcpclient/configuration/).

## Sessions

`factory.Session(ctx, name)` returns the `*mcp.ClientSession` to the server `name`, with the [MCP Go SDK](https://pkg.go.dev/github.com/modelcontextprotocol/go-sdk/mcp#ClientSession) API: `CallTool`, `ListTools`, `ReadResource`, `GetPrompt`…

* The session is opened on the first call, then reused: call `Session` every time you need it, it is cheap and safe for concurrent use.
* Server names are case-insensitive.
* The session is not tied to `ctx`: it outlives the request that opened it. `ctx` still traces the connection.
* If the connection fails, the error is returned and the next call retries.
* All sessions are closed when the application stops. After that, `Session` returns `mcpclient.ErrClosed`.

The client sends `app.name` and `app.version` as its name and version.

## Custom transports

A server can be reached through another transport than streamable HTTP, for example a local server started as a subprocess over stdio. Register a `mcpclient.TransportFunc` for its name with `AsTransport`: it is called each time the session is opened, and wins over `mcpclient.servers.<name>.url`. It is either a value, or a constructor whose parameters are injected by fx:

```go
mcpclient.AsTransport("git", func(ctx context.Context) (mcp.Transport, error) {
  return &mcp.CommandTransport{Command: exec.Command("mcp-server-git")}, nil
})
```

The requests are traced, measured and logged like the others. [mcpclienttest](/oryn/mcpclient/testing/) uses it to serve servers in memory.

## Configuration

The `mcpclient.*` keys set the URL, headers and timeout of each server, the request logs and the methods excluded from telemetry. See [Configuration](/oryn/mcpclient/configuration/).

# Testing

> Test MCP clients against in-memory MCP servers, without network, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/), and serve the MCP servers in memory with `mcpclienttest`, without network. The client is the real one: instrumentation, logs and timeout included.

## Serve a server

`mcpclienttest.Serve(name, server)` makes the server `name` resolve to an in-memory `*mcp.Server`, built with the [MCP Go SDK](https://pkg.go.dev/github.com/modelcontextprotocol/go-sdk/mcp):

```go
func TestSearch(t *testing.T) {
  t.Parallel()


  server := mcp.NewServer(&mcp.Implementation{Name: "docs", Version: "test"}, nil)
  mcp.AddTool(server, &mcp.Tool{Name: "search"},
    func(ctx context.Context, req *mcp.CallToolRequest, in SearchInput) (*mcp.CallToolResult, any, error) {
      return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: "test_result"}}}, nil, nil
    })


  var (
    search *Search
    spans  *tracetest.InMemoryExporter
  )


  coretest.Start(t, core.New(mcpclient.Module, fx.Provide(NewSearch)),
    mcpclienttest.Serve("docs", server),
    fx.Populate(&search, &spans),
  )


  result, err := search.Search(t.Context(), "oryn")
  require.NoError(t, err)
  assert.Equal(t, "test_result", result.Content[0].(*mcp.TextContent).Text)


  spantest.HasSpan(t, spans,
    spantest.WithName("tools/call search"),
    spantest.WithKind(trace.SpanKindClient),
    spantest.WithAttributeValue(mcpclient.GenAIToolNameKey.String("search")),
  )
}
```

No `mcpclient.servers.docs.url` is needed: the in-memory server wins over the configuration. Its sessions are closed when the application stops.

## Serve the application server

`mcpclienttest.ServeContainer(name)` serves the `*mcp.Server` of the application, provided by [mcpserver](/oryn/mcpserver/overview/), to test an application calling itself, or the client and server instrumentation together:

```go
coretest.Start(t, core.New(mcpclient.Module, mcpserver.Module, mcpserver.AsCapability(NewSearchTool)),
  mcpclienttest.ServeContainer("self"),
  fx.Populate(&factory, &spans),
)


session, err := factory.Session(t.Context(), "self")
require.NoError(t, err)


_, err = session.CallTool(t.Context(), &mcp.CallToolParams{Name: "search", Arguments: map[string]any{"query": "oryn"}})
require.NoError(t, err)


client := spantest.Find(spans, spantest.WithName("tools/call search"), spantest.WithKind(trace.SpanKindClient))
spantest.HasSpan(t, spans,
  spantest.WithName("tools/call search"),
  spantest.WithKind(trace.SpanKindServer),
  spantest.WithParent(client[0].SpanContext.SpanID()),
)
```

## Over the network

To test the streamable HTTP transport, start an MCP server on `127.0.0.1:0` with [mcpserver](/oryn/mcpserver/testing/#over-the-network), and point the server URL to it:

```go
var server *mcpserver.HTTPServer


coretest.Start(t, serverApp,
  config.AsOptions(config.WithValues(map[string]any{"mcpserver.address": "127.0.0.1:0"})),
  mcpserver.Serve(),
  fx.Populate(&server),
)


coretest.Start(t, clientApp,
  config.AsOptions(config.WithValues(map[string]any{"mcpclient.servers.docs.url": "http://" + server.Addr + "/mcp"})),
  fx.Populate(&factory),
)
```

# Configuration

> Every mcpserver configuration key, with its default and when to change it.

The `mcpserver.*` keys configure the MCP server and its streamable HTTP transport. All of them are optional.

## Reference

```yaml
mcpserver:
  address: ":8082"            # listen address
  path: /mcp                  # path of the MCP endpoint
  stateless: false            # no sessions: every request is independent
  json_response: false        # answer with application/json instead of SSE streams
  session_timeout: 30m        # idle sessions are closed after this duration, 0 = never
  read_header_timeout: 10s    # max duration to read the request headers
  log:
    enabled: true             # log every request
  observability:
    exclude: []               # MCP methods excluded from spans, metrics and logs
```

| Key                               | Type     | Default | Environment variable                |
| --------------------------------- | -------- | ------- | ----------------------------------- |
| `mcpserver.address`               | string   | `:8082` | `APP_MCPSERVER_ADDRESS`             |
| `mcpserver.path`                  | string   | `/mcp`  | `APP_MCPSERVER_PATH`                |
| `mcpserver.stateless`             | bool     | `false` | `APP_MCPSERVER_STATELESS`           |
| `mcpserver.json_response`         | bool     | `false` | `APP_MCPSERVER_JSON_RESPONSE`       |
| `mcpserver.session_timeout`       | duration | `30m`   | `APP_MCPSERVER_SESSION_TIMEOUT`     |
| `mcpserver.read_header_timeout`   | duration | `10s`   | `APP_MCPSERVER_READ_HEADER_TIMEOUT` |
| `mcpserver.log.enabled`           | bool     | `true`  | `APP_MCPSERVER_LOG_ENABLED`         |
| `mcpserver.observability.exclude` | list     | `[]`    |                                     |

The server name and version sent to the clients are [`app.name` and `app.version`](/oryn/core/configuration/#keys).

## Address and path

The MCP endpoint is `http://<address><path>`, `http://localhost:8082/mcp` by default:

```yaml
mcpserver:
  address: ":9000"
  path: /agents/mcp
```

```shell
claude mcp add --transport http my-app http://localhost:9000/agents/mcp
```

Once `mcpserver.Serve()` has started the server, the `*mcpserver.HTTPServer` Addr holds the real listening address, which is useful with port `0`.

## Sessions

By default, the server is stateful: a client opens a session (`Mcp-Session-Id` header), and the server keeps it until the client closes it, or until it has been idle for `session_timeout`. With `session_timeout: 0s`, idle sessions are never closed.

Stateful sessions are kept in memory: behind a load balancer, a client must reach the same instance. For several instances without sticky sessions, run stateless:

```yaml
mcpserver:
  stateless: true
```

In stateless mode, every request is independent: the server cannot send requests or notifications to the client, which is fine for tools, resources and prompts.

## Responses

By default, responses are sent as [server-sent event](https://html.spec.whatwg.org/multipage/server-sent-events.html) streams, which allow progress notifications. Some proxies buffer streams: answer with plain JSON instead:

```yaml
mcpserver:
  json_response: true
```

## Request logs

A log is written for every MCP request, `mcp request`, see [Observability](/oryn/mcpserver/observability/#logs). Disable it when the traces are enough:

```yaml
mcpserver:
  log:
    enabled: false
```

## Exclude methods

Exclude MCP methods from spans, metrics and request logs, for example the keep-alive pings and the discovery calls:

```yaml
mcpserver:
  observability:
    exclude:
      - ping
      - tools/list
```

Methods are the MCP method names: `initialize`, `tools/list`, `tools/call`, `resources/read`, `prompts/get`…

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the MCP server, following the MCP semantic conventions, and how to exclude methods.

Every MCP request goes through, from the outermost:

1. the OpenTelemetry instrumentation, following the [MCP semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/mcp/),
2. the request log,
3. the panic recovery,
4. the tool, resource or prompt handler.

With the streamable HTTP transport, the HTTP requests are also instrumented with [otelhttp](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp): a `POST /mcp` server span, and the `http.server.*` metrics.

## Traces

Each MCP request gets a server span, child of the `POST /mcp` HTTP span:

* named `{mcp.method.name} {target}`: `tools/call greet`, `resources/read docs://readme`, `prompts/get review`, or just the method without target: `tools/list`,
* with the attributes:

| Attribute               | Example                          | When             |
| ----------------------- | -------------------------------- | ---------------- |
| `mcp.method.name`       | `tools/call`                     | always           |
| `mcp.session.id`        | `4XZ…`                           | with a session   |
| `gen_ai.operation.name` | `execute_tool`                   | tool calls       |
| `gen_ai.tool.name`      | `greet`                          | tool calls       |
| `gen_ai.prompt.name`    | `review`                         | `prompts/get`    |
| `mcp.resource.uri`      | `docs://readme`                  | `resources/read` |
| `error.type`            | `tool_error`, `-32602`, `_OTHER` | failures         |

* with an error status when the request fails: `error.type` is the JSON-RPC error code for protocol errors (unknown tool, invalid params…), `_OTHER` for other errors (a panic…), and `tool_error` for a tool returning an error result.

When the client sends a trace context in the request `_meta` (`traceparent`, `tracestate`), as advised by the conventions, the span continues the trace of the client instead. Spans created in handlers with `ctx` are children of the MCP span.

## Metrics

| Metric                          | Description                              | Attributes                                                                |
| ------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------- |
| `mcp.server.operation.duration` | Duration of the MCP requests, in seconds | `mcp.method.name`, `gen_ai.tool.name`, `gen_ai.prompt.name`, `error.type` |

The instrumentation scope of the spans and metrics is `github.com/go-oryn/oryn/mcpserver`.

## Logs

Each request is logged as `mcp request`, at error level for failures and tool errors, and info level otherwise, with:

| Attribute    | Example                        |
| ------------ | ------------------------------ |
| `method`     | `tools/call`                   |
| `target`     | `greet`                        |
| `session_id` | `4XZ…`                         |
| `duration`   | `1.2ms`                        |
| `error_type` | `tool_error` (failures only)   |
| `error`      | `missing name` (failures only) |

The log is correlated with the request span. Disable request logs with `mcpserver.log.enabled: false`. The logs of the MCP SDK itself (sessions connecting…) are logged at debug level.

## Exclude methods

Noisy methods can be excluded from spans, metrics and request logs:

```yaml
mcpserver:
  observability:
    exclude:
      - ping
      - notifications/initialized
```

# MCP server

> The mcpserver module, a Model Context Protocol server with tools, resources and prompts registered through fx, served over streamable HTTP.

The `mcpserver` module provides a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, built on the official [MCP Go SDK](https://github.com/modelcontextprotocol/go-sdk), instrumented with OpenTelemetry. It lets AI agents call the tools, read the resources and get the prompts of your application.

```shell
go get github.com/go-oryn/oryn/mcpserver
```

## Quick start

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    mcpserver.Module,
    mcpserver.AsCapability(NewGreetTool),
  ).Run(mcpserver.Serve())
}
```

* `mcpserver.Module` provides the `*mcp.Server`, with the registered capabilities, and the `*mcpserver.HTTPServer` serving it.
* `mcpserver.AsCapability` registers tools, resources and prompts, see [Capabilities](#capabilities).
* `mcpserver.Serve()` starts the [streamable HTTP transport](https://modelcontextprotocol.io/specification/latest/basic/transports#streamable-http) with the application, on `:8082/mcp`, and shuts it down gracefully when the application stops. If the server fails while running, the application stops.
* Every request is traced, measured and logged, see [Observability](/oryn/mcpserver/observability/).

The `*mcpserver.HTTPServer` wraps an `*http.Server`: it is a dedicated type, so the MCP server can run next to the [HTTP server](/oryn/httpserver/overview/) in the same application.

## Capabilities

A capability registers tools, resources or prompts on the `*mcp.Server`, with the [MCP Go SDK](https://pkg.go.dev/github.com/modelcontextprotocol/go-sdk/mcp) API. It is either:

* a value: a `mcpserver.Capability` (with a `Register(*mcp.Server) error` method), or a `func(*mcp.Server) error`,
* or a constructor returning one, whose parameters are injected by fx.

Capabilities are registered in registration order. If one fails, the application fails to start.

### Tools

`mcp.AddTool` adds a typed tool: the input and output JSON schemas are inferred from the Go types, and the input is validated and decoded before the handler is called.

```go
type GreetInput struct {
  Name string `json:"name" jsonschema:"the name to greet"`
}


type GreetOutput struct {
  Greeting string `json:"greeting"`
}


// GreetTool greets with the greeting of the config.
type GreetTool struct {
  greeting string
}


func NewGreetTool(cfg *config.Config) *GreetTool {
  return &GreetTool{greeting: cfg.GetStringOrDefault("greet.greeting", "hello")}
}


func (t *GreetTool) Register(server *mcp.Server) error {
  mcp.AddTool(server, &mcp.Tool{Name: "greet", Description: "Greets someone"}, t.greet)


  return nil
}


func (t *GreetTool) greet(ctx context.Context, req *mcp.CallToolRequest, in GreetInput) (*mcp.CallToolResult, GreetOutput, error) {
  if in.Name == "" {
    // Returned to the agent as a tool error result.
    return nil, GreetOutput{}, errors.New("missing name")
  }


  return nil, GreetOutput{Greeting: t.greeting + " " + in.Name}, nil
}
```

```go
mcpserver.AsCapability(NewGreetTool)
```

An error returned by a typed tool handler is sent to the agent as a tool error result (`isError: true`), so it can correct its call.

### Resources and prompts

```go
func NewDocs(repository *docs.Repository) mcpserver.CapabilityFunc {
  return func(server *mcp.Server) error {
    server.AddResource(&mcp.Resource{URI: "docs://readme", Name: "readme", MIMEType: "text/markdown"},
      func(ctx context.Context, req *mcp.ReadResourceRequest) (*mcp.ReadResourceResult, error) {
        return &mcp.ReadResourceResult{Contents: []*mcp.ResourceContents{
          {URI: req.Params.URI, MIMEType: "text/markdown", Text: repository.Readme(ctx)},
        }}, nil
      })


    server.AddPrompt(&mcp.Prompt{Name: "review", Description: "Reviews a document"},
      func(ctx context.Context, req *mcp.GetPromptRequest) (*mcp.GetPromptResult, error) {
        return &mcp.GetPromptResult{Messages: []*mcp.PromptMessage{
          {Role: "user", Content: &mcp.TextContent{Text: "Review the readme."}},
        }}, nil
      })


    return nil
  }
}
```

A panic in a handler is recovered, logged with its stack, and answered with an error.

## Connect an agent

Point an MCP client to the endpoint, for example with Claude Code:

```shell
claude mcp add --transport http my-app http://localhost:8082/mcp
```

Or inspect the server with the [MCP Inspector](https://github.com/modelcontextprotocol/inspector):

```shell
npx @modelcontextprotocol/inspector
```

## Configuration

The `mcpserver.*` keys set the address and path, the session mode, the response format, the request logs and the methods excluded from telemetry. See [Configuration](/oryn/mcpserver/configuration/).

# Testing

> Test MCP tools, resources and prompts in memory, with a real MCP client, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/), and connect an MCP client to the `*mcp.Server` with `mcpservertest.Connect`, through in-memory transports, without network:

```go
func TestGreet(t *testing.T) {
  t.Parallel()


  var (
    server *mcp.Server
    spans  *tracetest.InMemoryExporter
    logs   *logtest.InMemoryExporter
  )


  coretest.Start(t, core.New(mcpserver.Module, mcpserver.AsCapability(NewGreetTool)), fx.Populate(&server, &spans, &logs))


  session := mcpservertest.Connect(t, server)


  result, err := session.CallTool(t.Context(), &mcp.CallToolParams{
    Name:      "greet",
    Arguments: map[string]any{"name": "oryn"},
  })
  require.NoError(t, err)
  assert.False(t, result.IsError)


  spantest.HasSpan(t, spans,
    spantest.WithName("tools/call greet"),
    spantest.WithAttributeValue(mcpserver.GenAIToolNameKey.String("greet")),
  )
  logtest.HasRecord(t, logs, logtest.WithBody("mcp request"), logtest.WithAttributeValue("target", "greet"))
}
```

The request goes through the whole server: instrumentation, logs, recovery and handler. The sessions are closed when the test ends.

## Over the network

To test the streamable HTTP transport, set `mcpserver.address` to `127.0.0.1:0` and add `mcpserver.Serve()`: once started, the `*mcpserver.HTTPServer` Addr is the listening address.

```go
var server *mcpserver.HTTPServer


coretest.Start(t, app,
  config.AsOptions(config.WithValues(map[string]any{"mcpserver.address": "127.0.0.1:0"})),
  mcpserver.Serve(),
  fx.Populate(&server),
)


client := mcp.NewClient(&mcp.Implementation{Name: "test", Version: "test"}, nil)
session, err := client.Connect(t.Context(), &mcp.StreamableClientTransport{Endpoint: "http://" + server.Addr + "/mcp"}, nil)
```

# Configuration

> Every sql configuration key, with its default and when to change it.

The `sql.*` keys configure the database connection pool. `sql.driver` and `sql.dsn` are required.

## Reference

```yaml
sql:
  driver: postgres            # postgres, mysql or sqlite (required)
  dsn: postgres://app:${DATABASE_PASSWORD}@localhost:5432/app   # data source name (required)
  max_open_conns: 0           # max open connections, 0 = unlimited
  max_idle_conns: 2           # max idle connections kept in the pool, 0 = none
  conn_max_lifetime: 0s       # max lifetime of a connection, 0 = unlimited
  conn_max_idle_time: 0s      # max idle time of a connection, 0 = unlimited
  ping_on_start: true         # fail the application start if the database is unreachable
  observability:
    query: true               # record the query text in the spans
```

| Key                       | Type     | Default          | Environment variable          |
| ------------------------- | -------- | ---------------- | ----------------------------- |
| `sql.driver`              | string   | required         | `APP_SQL_DRIVER`              |
| `sql.dsn`                 | string   | required         | `APP_SQL_DSN`                 |
| `sql.max_open_conns`      | int      | `0` (unlimited)  | `APP_SQL_MAX_OPEN_CONNS`      |
| `sql.max_idle_conns`      | int      | `2` (`0` = none) | `APP_SQL_MAX_IDLE_CONNS`      |
| `sql.conn_max_lifetime`   | duration | `0s` (unlimited) | `APP_SQL_CONN_MAX_LIFETIME`   |
| `sql.conn_max_idle_time`  | duration | `0s` (unlimited) | `APP_SQL_CONN_MAX_IDLE_TIME`  |
| `sql.ping_on_start`       | bool     | `true`           | `APP_SQL_PING_ON_START`       |
| `sql.observability.query` | bool     | `true`           | `APP_SQL_OBSERVABILITY_QUERY` |

## Driver and DSN

| `sql.driver` | DSN example                                                |
| ------------ | ---------------------------------------------------------- |
| `postgres`   | `postgres://app:secret@localhost:5432/app?sslmode=disable` |
| `mysql`      | `app:secret@tcp(localhost:3306)/app?parseTime=true`        |
| `sqlite`     | `app.db`, `file:app.db?_pragma=foreign_keys(1)`            |

See [Drivers](/oryn/sql/overview/#drivers) for the DSN formats.

Keep the password out of the files, with a [placeholder](/oryn/core/configuration/#environment-variable-placeholders) or the whole DSN in an environment variable:

```yaml
sql:
  dsn: postgres://app:${DATABASE_PASSWORD}@db:5432/app
```

```shell
APP_SQL_DSN="postgres://app:secret@db:5432/app" go run ./cmd/serve
```

A placeholder can have a default: `${DATABASE_PASSWORD:local}`.

## Pool

The defaults are those of [database/sql](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns). In production, bound the pool to protect the database, and recycle the connections so they follow failovers and load balancer changes:

```yaml
sql:
  max_open_conns: 20
  max_idle_conns: 10
  conn_max_lifetime: 30m
  conn_max_idle_time: 5m
```

Size `max_open_conns` with the number of instances: their sum must stay below the database connection limit. The pool usage is measured, see [Observability](/oryn/sql/observability/#metrics).

## Ping on start

By default, the application pings the database when it starts, and fails to start if it is unreachable: a misconfigured DSN is found at deployment, not at the first request. To start anyway, for example when the database may come up later, disable it:

```yaml
sql:
  ping_on_start: false
```

The readiness [health probe](/oryn/core/health/) still reports the database state.

## Query text

The query spans record the SQL text in `db.query.text`, with its placeholders, never the arguments. If the queries themselves are sensitive, do not record them:

```yaml
sql:
  observability:
    query: false
```

## Per environment

```yaml
# configs/app.yaml: local development, SQLite
sql:
  driver: sqlite
  dsn: app.db


# configs/prod/sql.yaml (APP_ENV=prod): PostgreSQL
sql:
  driver: postgres
  dsn: postgres://app:${DATABASE_PASSWORD}@db:5432/app
  max_open_conns: 20
  conn_max_lifetime: 30m
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Migrations

> Apply SQL migrations when the application starts, with goose.

The `sql` module applies migrations with [goose](https://pressly.github.io/goose/): SQL files, numbered, applied in order, each once.

## Write migrations

Put the migration files in a directory, and embed it:

```plaintext
migrations/
├── 00001_create_users.sql
├── 00002_add_users_email.sql
└── embed.go
```

```sql
-- migrations/00001_create_users.sql


-- +goose Up
CREATE TABLE users (
    id   BIGSERIAL PRIMARY KEY,
    name TEXT NOT NULL
);


-- +goose Down
DROP TABLE users;
```

migrations/embed.go

```go
package migrations


import "embed"


//go:embed *.sql
var FS embed.FS
```

See the [goose annotations](https://pressly.github.io/goose/documentation/annotations/) for statements spanning several lines, or running outside a transaction.

## Apply them on start

Register the migrations with `sql.AsMigrations`, and pass `sql.Migrate()` to `Run`, **before** the servers:

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    sql.Module,
    sql.AsMigrations(migrations.FS),
    httpserver.Module,
    // ...
  ).Run(sql.Migrate(), httpserver.Serve())
}
```

When the application starts, the pending migrations are applied, in order, before the servers start: requests never hit an outdated schema. Applied migrations are recorded in the `goose_db_version` table. The application fails to start if a migration fails, and the `database migrated` log tells how many were applied. The migration queries are traced under a single `sql.migrate` span, with a `sql.migrations.applied` attribute.

* Register one migrations file system, with the files at its root.
* With several instances of the application, migrations may run concurrently: prefer a dedicated command, see below.

## Apply them with a command

`RunOnce` starts the application then stops it: a command applying the migrations, for a deployment job or an init container, is one line.

cmd/migrate/main.go

```go
func main() {
  app.App.RunOnce(sql.Migrate())
}
```

```shell
go run ./cmd/migrate
```

Only the database pool is built: the servers and handlers of the application are not. See [Several entrypoints](/oryn/core/application/#several-entrypoints), and the [demo](/oryn/demo/entrypoints/).

# Observability

> The traces and metrics of the sql module, and the migration span.

The pool is instrumented with [otelsql](https://github.com/XSAM/otelsql), with the providers of the application.

## Traces

A client span per query, named after the operation (`SELECT`, `INSERT`…), with `db.system.name`, `db.operation.name` and the query text in `db.query.text`. Transactions get `sql.conn.begin_tx`, `sql.tx.commit`… spans.

The query text holds the SQL with its placeholders, not the arguments. Disable it with `sql.observability.query: false`.

## Metrics

| Metric                                                                            | Description     |
| --------------------------------------------------------------------------------- | --------------- |
| `db.client.operation.duration`                                                    | Query duration  |
| `db.sql.connection.open`, `db.sql.connection.max_open`, `db.sql.connection.wait`… | Pool statistics |

## Migrations

`sql.Migrate()` traces the migration queries under one `sql.migrate` span, with the number of applied migrations in `sql.migrations.applied`, and logs `database migrated` with `applied=N`. See [Migrations](/oryn/sql/migrations/).

## Your own spans

The query spans are children of the span in the context: pass the request context to the `...Context` methods (`QueryContext`, `ExecContext`…), and wrap a business operation in a span of your own to group its queries:

```go
func (s *Service) Create(ctx context.Context, title string) (Task, error) {
  ctx, span := s.telemetry.Tracer().Start(ctx, "tasks.create")
  defer span.End()


  // The INSERT span is a child of tasks.create.
  _, err := s.db.ExecContext(ctx, "INSERT INTO tasks (id, title) VALUES ($1, $2)", id, title)
  ...
}
```

# SQL

> The sql module, a database/sql connection pool to PostgreSQL, MySQL or SQLite, instrumented with OpenTelemetry.

The `sql` module provides a standard [`*sql.DB`](https://pkg.go.dev/database/sql#DB) connection pool to PostgreSQL, MySQL or SQLite, instrumented with OpenTelemetry, with a health probe and [migrations](/oryn/sql/migrations/).

```shell
go get github.com/go-oryn/oryn/sql
```

## Quick start

configs/app.yaml

```yaml
sql:
  driver: postgres
  dsn: postgres://app:${DATABASE_PASSWORD}@localhost:5432/app
```

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    sql.Module,
    fx.Provide(NewUserRepository),
  ).Run()
}
```

The `*sql.DB` is injected like any dependency. Always pass the context, so queries are traced as children of the current span:

```go
import "database/sql"


type UserRepository struct {
  db *sql.DB
}


func NewUserRepository(db *sql.DB) *UserRepository {
  return &UserRepository{db: db}
}


func (r *UserRepository) Find(ctx context.Context, id int64) (User, error) {
  var user User


  err := r.db.QueryRowContext(ctx, "SELECT id, name FROM users WHERE id = $1", id).Scan(&user.ID, &user.Name)


  return user, err
}
```

The oryn package is named `sql` like `database/sql`: in the files using both, import it with an alias, e.g. `orynsql "github.com/go-oryn/oryn/sql"`.

## Drivers

| `sql.driver` | Database       | Driver                                                                       | DSN example                                                   |
| ------------ | -------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------- |
| `postgres`   | PostgreSQL     | [pgx](https://github.com/jackc/pgx)                                          | `postgres://user:password@localhost:5432/app?sslmode=disable` |
| `mysql`      | MySQL, MariaDB | [go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)                | `user:password@tcp(localhost:3306)/app?parseTime=true`        |
| `sqlite`     | SQLite         | [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite), pure Go, no cgo | `app.db`, `file:app.db?_pragma=foreign_keys(1)`               |

The drivers are included in the module: there is nothing else to import. Query placeholders are those of the database: `$1` for PostgreSQL, `?` for MySQL and SQLite.

## Lifecycle

* When the application starts, the database is pinged: the application fails to start if it is unreachable. Disable it with `sql.ping_on_start: false`.
* The module registers a readiness and startup [health probe](/oryn/core/health/) named `sql`, pinging the database. It is not a liveness probe: a database outage must not restart the application.
* The pool is closed when the application stops, after the servers.

## Observability

Every query is traced and measured, see [Observability](/oryn/sql/observability/).

## Configuration

The `sql.*` keys set the database, the pool, the ping on start and the query text in spans. See [Configuration](/oryn/sql/configuration/).

# Testing

> Test database code with SQLite, migrations, and the query spans.

Tests can run against a real database, or against SQLite, which needs no server.

## SQLite in a temporary file

Use a SQLite database in the test temporary directory: each test gets its own, deleted after the test.

```go
func TestUserRepository(t *testing.T) {
  t.Parallel()


  var (
    repository *UserRepository
    spans      *tracetest.InMemoryExporter
  )


  coretest.Start(t,
    core.New(sql.Module, sql.AsMigrations(migrations.FS), fx.Provide(NewUserRepository)),
    config.AsOptions(config.WithValues(map[string]any{
      "sql.driver": "sqlite",
      "sql.dsn":    filepath.Join(t.TempDir(), "test.db"),
    })),
    sql.Migrate(),
    fx.Populate(&repository, &spans),
  )


  user, err := repository.Create(t.Context(), "test_user")
  require.NoError(t, err)


  spantest.HasSpan(t, spans, spantest.WithName("INSERT"))
}
```

* `sql.Migrate()` applies the migrations before the test runs, like in production.
* Prefer a file over `:memory:`: an in-memory SQLite database exists per connection, and the pool opens several.
* SQL must then work with both databases: placeholders (`?` works in SQLite and MySQL, `$1` in SQLite and PostgreSQL) and types differ. When they do not, test against the real database.

## A real database

Set `sql.driver` and `sql.dsn` to a database started for the tests, for example with [Testcontainers](https://golang.testcontainers.org/) or a CI service, and skip the tests when it is not available:

```go
dsn := os.Getenv("TEST_POSTGRES_DSN")
if dsn == "" {
  t.Skip("TEST_POSTGRES_DSN is not set")
}
```

# Configuration

> Every validator configuration key, with its default and when to change it.

The `validator.*` keys configure the validator. All of them are optional.

## Reference

```yaml
validator:
  field_name_tag: json        # struct tag naming the fields in the errors
  observability:
    enabled: true             # record the failures on the span, and count them
```

| Key                               | Type   | Default | Environment variable                  |
| --------------------------------- | ------ | ------- | ------------------------------------- |
| `validator.field_name_tag`        | string | `json`  | `APP_VALIDATOR_FIELD_NAME_TAG`        |
| `validator.observability.enabled` | bool   | `true`  | `APP_VALIDATOR_OBSERVABILITY_ENABLED` |

## Field names

The field errors name the fields after their `json` tag, so they match the request the client sent. Fields without the tag, or with `json:"-"`, use their Go name.

For forms decoded from a `form` tag, name the fields after it instead:

```yaml
validator:
  field_name_tag: form
```

```go
type SignupForm struct {
  Email string `form:"email_address" validate:"required,email"`
}
// field error: {"field": "email_address", ...}
```

## Observability

Failures are recorded on the active span and counted, see [Observability](/oryn/validator/observability/). Disable it if validation failures are not worth measuring, for example for an internal tool:

```yaml
validator:
  observability:
    enabled: false
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> How the validator records the validation failures on the active span and counts them.

Validations are fast and frequent: the validator creates no span. When a validation fails, it records the failure on the span of the context given to `Struct` or `Var`, usually the span of the HTTP request, and counts it.

The values of the fields are never recorded: they may be sensitive.

## Traces

A failed validation adds a `validation failed` event to the active span:

| Attribute           | Description                                                        |
| ------------------- | ------------------------------------------------------------------ |
| `validation.type`   | Validated type, like `users.CreateUserRequest`, or `var` for `Var` |
| `validation.fields` | Invalid fields, like `["email", "address.city"]`                   |

```plaintext
POST /users                                     server span
  event: validation failed
    validation.type   = users.CreateUserRequest
    validation.fields = [email address.city]
```

Without a recording span in the context, nothing is added.

## Metrics

| Metric                | Type    | Description                        |
| --------------------- | ------- | ---------------------------------- |
| `validation.failures` | counter | Number of failed field validations |

Attributes: `validation.type` and `validation.tag`, the failed tag (`required`, `email`…). A request with two invalid fields counts two failures.

```promql
# Most frequent failures, per type and tag
sum by (validation_type, validation_tag) (rate(validation_failures_total[5m]))
```

The metrics are emitted with the instrumentation scope `github.com/go-oryn/oryn/validator`, with the meter provider of the application.

## Disable

```yaml
validator:
  observability:
    enabled: false
```

# Validator

> The validator module, struct and value validation with go-playground/validator, custom validations registered through fx, and field errors ready for API responses.

The `validator` module validates structs and values with the rules of their `validate` tags, using [go-playground/validator](https://github.com/go-playground/validator), and returns field errors ready to send in an API response. Validation failures are recorded on the active span and counted.

```shell
go get github.com/go-oryn/oryn/validator
```

## Quick start

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    httpserver.Module,
    validator.Module,
    httpserver.AsHandler("POST /users", NewCreateUserHandler),
  ).Run(httpserver.Serve())
}
```

* `validator.Module` provides the `*validator.Validator`, and the underlying go-playground `*validator.Validate`.
* Custom validations, struct validations and aliases are registered with `AsValidation`, `AsStructValidation` and `AsAlias`, see [Custom validations](#custom-validations).
* Failures are recorded on the span of the context, see [Observability](/oryn/validator/observability/).

## Validate a request

Describe the rules with the [go-playground tags](https://pkg.go.dev/github.com/go-playground/validator/v10#readme-baked-in-validations), then call `Struct` with the request context:

```go
type CreateUserRequest struct {
  Name    string   `json:"name"    validate:"required,max=100"`
  Email   string   `json:"email"   validate:"required,email"`
  Role    string   `json:"role"    validate:"oneof=admin member"`
  Address Address  `json:"address"`
  Tags    []string `json:"tags"    validate:"max=5,dive,min=1"`
}


type Address struct {
  City string `json:"city" validate:"required"`
}


func NewCreateUserHandler(v *validator.Validator, users *UserService) http.HandlerFunc {
  return func(w http.ResponseWriter, r *http.Request) {
    var request CreateUserRequest
    if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
      http.Error(w, "invalid JSON", http.StatusBadRequest)
      return
    }


    if err := v.Struct(r.Context(), request); err != nil {
      var errs validator.ValidationErrors
      if errors.As(err, &errs) {
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusBadRequest)
        _ = json.NewEncoder(w).Encode(map[string]any{"errors": errs})
        return
      }


      http.Error(w, err.Error(), http.StatusInternalServerError)
      return
    }


    ...
  }
}
```

```json
{
  "errors": [
    {"field": "email", "tag": "email", "message": "must be a valid email address"},
    {"field": "address.city", "tag": "required", "message": "is required"},
    {"field": "tags", "tag": "max", "param": "5", "message": "must be at most 5 items long"}
  ]
}
```

* `Struct(ctx, s)` validates a struct, or a pointer to one: nested structs are validated too. Required structs are enabled: `validate:"required"` on a struct field fails on its zero value.
* `Var(ctx, value, tags)` validates a single value: `v.Var(ctx, email, "required,email")`. Its field errors have no field.
* Both return `validator.ValidationErrors` when a field is invalid, and another error when the input cannot be validated (a nil pointer, a value that is not a struct): answer `500` to those.

`ValidationErrors` is a list of `FieldError`:

| Field     | Description                                                                                                |
| --------- | ---------------------------------------------------------------------------------------------------------- |
| `Field`   | Path of the field, named after its `json` tag, without the struct name: `email`, `address.city`, `tags[2]` |
| `Tag`     | Failed tag: `required`, `email`, `max`…                                                                    |
| `Param`   | Parameter of the tag, like `100` for `max=100`                                                             |
| `Message` | English message: `is required`, `must be at most 100 characters long`…                                     |

`errs.Fields()` returns a map of field to message, convenient for forms. Unknown tags get a generic message, `must satisfy <tag>`.

## Custom validations

### Field validations

`AsValidation` registers a tag. The validation is a go-playground `Func` or `FuncCtx`, or a constructor returning one, whose dependencies are injected by fx. Import go-playground with an alias, `playground "github.com/go-playground/validator/v10"`, in the files declaring validations:

```go
// A value: no dependency.
validator.AsValidation("slug", func(fl playground.FieldLevel) bool {
  return slugPattern.MatchString(fl.Field().String())
})


// A constructor: the validation uses a repository, and the context of the validation.
func NewUniqueEmailValidation(users *UserRepository) playground.FuncCtx {
  return func(ctx context.Context, fl playground.FieldLevel) bool {
    exists, err := users.EmailExists(ctx, fl.Field().String())
    return err == nil && !exists
  }
}


validator.AsValidation("unique_email", NewUniqueEmailValidation)
```

```go
type CreateUserRequest struct {
  Email string `json:"email" validate:"required,email,unique_email"`
  Slug  string `json:"slug"  validate:"required,slug"`
}
```

### Struct validations

A struct validation, a `StructLevelFunc` or `StructLevelFuncCtx` or a constructor returning one, checks fields together. It reports its errors with `ReportError(value, fieldName, structFieldName, tag, param)`:

```go
func validatePeriod(sl playground.StructLevel) {
  period := sl.Current().Interface().(Period)
  if period.End.Before(period.Start) {
    sl.ReportError(period.End, "end", "End", "after_start", "")
  }
}


validator.AsStructValidation(validatePeriod, Period{})
```

It runs after the field validations of the struct. A type has one struct validation: the last registered one wins.

### Aliases

An alias names a list of tags:

```go
validator.AsAlias("username", "required,min=3,max=30,alphanum")
```

A tag registered twice, by a validation or an alias, or a reserved tag (`omitempty`, `dive`…), fails the application start.

## Configuration

The `validator.*` keys set the tag naming the fields, and the observability. See [Configuration](/oryn/validator/configuration/).

# Testing

> Test validation rules and custom validations in unit tests, and validation failures through the application.

## Unit tests

`validatortest.New` returns a `*validator.Validator` built like in the application, with your registrations, without writing an application:

```go
func TestCreateUserRequest(t *testing.T) {
  t.Parallel()


  v := validatortest.New(t, validator.AsAlias("username", "required,min=3,max=30,alphanum"))


  err := v.Struct(t.Context(), CreateUserRequest{Name: "a", Email: "invalid"})


  var errs validator.ValidationErrors
  require.ErrorAs(t, err, &errs)
  assert.Equal(t, map[string]string{
    "name":  "must be at least 3 characters long",
    "email": "must be a valid email address",
  }, errs.Fields())
}
```

A custom validation with dependencies gets them from the options, like in the application. Replace a real dependency with a fake:

```go
v := validatortest.New(t,
  fx.Supply(fakeUserRepository{existing: "taken@example.com"}),
  validator.AsValidation("unique_email", NewUniqueEmailValidation),
)


require.Error(t, v.Var(t.Context(), "taken@example.com", "unique_email"))
require.NoError(t, v.Var(t.Context(), "free@example.com", "unique_email"))
```

## Through the application

Test the handler with [coretest](/oryn/core/testing/): the `400` response, and the `validation failed` event on the request span.

```go
func TestCreateUserInvalid(t *testing.T) {
  t.Parallel()


  var (
    server  *http.Server
    spans   *tracetest.InMemoryExporter
    metrics *sdkmetric.ManualReader
  )


  coretest.Start(t, core.New(httpserver.Module, validator.Module, users.Module), fx.Populate(&server, &spans, &metrics))


  recorder := httptest.NewRecorder()
  server.Handler.ServeHTTP(recorder, httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/users",
    strings.NewReader(`{"name": "test", "email": "invalid"}`)))


  assert.Equal(t, http.StatusBadRequest, recorder.Code)


  request := spantest.Find(spans, spantest.WithName("POST /users"))
  require.Len(t, request, 1)
  assert.Equal(t, validator.EventFailed, request[0].Events[0].Name)


  assert.Equal(t, int64(1), metrictest.Int64Sum(t, metrics, validator.MetricFailures, validator.TagKey.String("email")))
}
```

# Configuration

> Every worker configuration key, with its default and when to change it.

The `worker.*` keys configure the restart policy of the workers, globally and per worker. All of them are optional.

## Reference

```yaml
worker:
  shutdown_on_failure: true   # shut the application down when a worker fails
  restart:
    enabled: true             # restart the failed runs
    max_restarts: 5           # consecutive restarts before failing, 0 = unlimited
    backoff:
      initial: 1s             # delay before the first restart
      max: 30s                # max delay between restarts
    reset_after: 1m           # run duration resetting the consecutive restarts, 0 = never
  workers:
    <name>:                   # per worker, overrides the options set in code
      enabled: true           # run the worker
      replicas: 1             # concurrent replicas
      restart:                # same keys as worker.restart, defaulting to them
        enabled: true
        max_restarts: 5
        backoff:
          initial: 1s
          max: 30s
        reset_after: 1m
```

| Key                               | Type     | Default                              | Environment variable                  |
| --------------------------------- | -------- | ------------------------------------ | ------------------------------------- |
| `worker.shutdown_on_failure`      | bool     | `true`                               | `APP_WORKER_SHUTDOWN_ON_FAILURE`      |
| `worker.restart.enabled`          | bool     | `true`                               | `APP_WORKER_RESTART_ENABLED`          |
| `worker.restart.max_restarts`     | int      | `5` (`0` = unlimited)                | `APP_WORKER_RESTART_MAX_RESTARTS`     |
| `worker.restart.backoff.initial`  | duration | `1s`                                 | `APP_WORKER_RESTART_BACKOFF_INITIAL`  |
| `worker.restart.backoff.max`      | duration | `30s`                                | `APP_WORKER_RESTART_BACKOFF_MAX`      |
| `worker.restart.reset_after`      | duration | `1m` (`0s` = never)                  | `APP_WORKER_RESTART_RESET_AFTER`      |
| `worker.workers.<name>.enabled`   | bool     | `true`                               | `APP_WORKER_WORKERS_<NAME>_ENABLED`   |
| `worker.workers.<name>.replicas`  | int      | `1`, or `WithReplicas`               | `APP_WORKER_WORKERS_<NAME>_REPLICAS`  |
| `worker.workers.<name>.restart.*` |          | `worker.restart.*`, or `WithRestart` | `APP_WORKER_WORKERS_<NAME>_RESTART_*` |

Durations use the Go syntax: `500ms`, `30s`, `2m`. An explicit zero is a value: `max_restarts: 0` means unlimited restarts, it does not fall back to the default.

For a worker, the settings are resolved from the highest precedence:

1. the `worker.workers.<name>.*` keys,
2. the `WithReplicas` and `WithRestart` options set in code,
3. the `worker.restart.*` keys,
4. the defaults.

The application fails to start with less than 1 replica, a negative `max_restarts` or a negative duration. A `backoff.max` lower than `backoff.initial` is raised to it.

## Restarts

The backoff doubles at each consecutive restart: with the defaults, `1s`, `2s`, `4s`, `8s`, `16s`, then the worker fails. For a worker that must never give up, like a queue consumer waiting for its broker, allow unlimited restarts:

```yaml
worker:
  restart:
    max_restarts: 0
    backoff:
      max: 1m
```

To fail on the first error instead, and let the orchestrator restart the application:

```yaml
worker:
  restart:
    enabled: false
```

## Failures

By default, a failed worker shuts the application down with exit code 1. To keep the application running, and only report the failure with the [health probe](/oryn/worker/overview/#health) and the logs:

```yaml
worker:
  shutdown_on_failure: false
```

The readiness probe then fails, so the instance stops receiving traffic, and the liveness probe lets the orchestrator restart it.

## Per worker

The keys under `worker.workers.<name>` apply to the worker registered with that name, and override the options set in code:

```yaml
worker:
  workers:
    orders-consumer:
      replicas: 8
      restart:
        max_restarts: 0
    outbox:
      enabled: false
```

A disabled worker is not started, its state is `disabled`, and it is healthy.

## Per environment

configs/app.yaml

```yaml
worker:
  workers:
    orders-consumer:
      replicas: 8


# configs/dev/worker.yaml (APP_ENV=dev)
worker:
  workers:
    orders-consumer:
      replicas: 1
    outbox:
      enabled: false
```

See [Configuration](/oryn/core/configuration/) for the loading rules.

# Observability

> Traces, metrics and logs of the worker runs.

The module uses the tracer, meter and logger providers of the application, with the `github.com/go-oryn/oryn/worker` instrumentation scope.

## Traces

Each run of a worker replica gets its own root span, of kind internal:

* named `worker <name>`: `worker outbox`,
* with the `worker.name` and `worker.replica` attributes,
* with an error status, the recorded error and the `error.type` attribute for a failed run: the Go type of the error (`*net.OpError`), or `panic`.

A run stopped by the application stopping is not an error. The context passed to `Run` carries the span: the spans created by the worker are its children, and its logs are correlated with it.

For a worker processing items in a loop, the span of a run can last for the whole life of the application. Create a span per item in the worker, as a child of the run span or as a new root, linked to the message for a queue consumer.

## Metrics

| Metric            | Type            | Unit        | Attributes                                | Description                       |
| ----------------- | --------------- | ----------- | ----------------------------------------- | --------------------------------- |
| `worker.runs`     | counter         | `{run}`     | `worker.name`, `error.type` (failed runs) | Number of worker runs, by outcome |
| `worker.restarts` | counter         | `{restart}` | `worker.name`                             | Number of worker restarts         |
| `worker.active`   | up-down counter | `{replica}` | `worker.name`                             | Number of running worker replicas |

## Logs

| Log                 | Level | Attributes                                                                        |
| ------------------- | ----- | --------------------------------------------------------------------------------- |
| `worker started`    | info  | `worker`, `replica`                                                               |
| `worker stopped`    | info  | `worker`, `replica`                                                               |
| `worker restarting` | warn  | `worker`, `replica`, the error, `restart` (consecutive restart number), `backoff` |
| `worker failed`     | error | `worker`, `replica`, the error                                                    |
| `worker panic`      | error | `worker`, `replica`, `panic`, `stack`                                             |
| `worker disabled`   | info  | `worker`                                                                          |

The run logs are correlated with the run span.

# Worker

> The worker module, long-running background workers run with the application, restarted when they fail, and reported by a health probe.

The `worker` module runs long-running background workers with the application: queue consumers, pollers, outbox relays… Each worker runs in its own goroutines, is restarted with a backoff when it fails or panics, is reported by a [health probe](/oryn/core/health/), and is instrumented with OpenTelemetry.

```shell
go get github.com/go-oryn/oryn/worker
```

## Quick start

```go
func main() {
  core.New(
    core.WithConfigFS(configs.FS),
    worker.Module,
    worker.AsWorker("outbox", NewOutboxWorker),
  ).Run(worker.Run())
}
```

* `worker.Module` provides the `*worker.Pool` of the registered workers, and registers its liveness and readiness probe.
* `worker.Run()` starts the workers with the application, and stops them when the application stops.
* Every run is traced, measured and logged, see [Observability](/oryn/worker/observability/).

## Workers

A worker implements `worker.Worker`:

```go
type Worker interface {
  Run(ctx context.Context) error
}
```

`Run` blocks until the work is done, or until `ctx` is canceled when the application stops; it should then return promptly. A typical worker loops until its context is canceled:

```go
type OutboxWorker struct {
  repository *Repository
  publisher  *Publisher
}


func NewOutboxWorker(repository *Repository, publisher *Publisher) *OutboxWorker {
  return &OutboxWorker{repository: repository, publisher: publisher}
}


func (w *OutboxWorker) Run(ctx context.Context) error {
  ticker := time.NewTicker(time.Second)
  defer ticker.Stop()


  for {
    select {
    case <-ctx.Done():
      return nil
    case <-ticker.C:
      if err := w.relay(ctx); err != nil {
        return err // restarted with a backoff
      }
    }
  }
}
```

Register it under a unique name, used in the config keys, the logs and the telemetry. The worker is either a value (a `worker.Worker`, a `worker.WorkerFunc` or a `func(context.Context) error`), or a constructor returning one, whose parameters are injected by fx:

```go
worker.AsWorker("outbox", NewOutboxWorker)
worker.AsWorker("cleanup", func(ctx context.Context) error { ... })
```

The application fails to start if the worker is not a worker or a constructor returning one, or if a name is registered twice.

## Run outcomes

| `Run`…                                 | Then the replica is                                                     |
| -------------------------------------- | ----------------------------------------------------------------------- |
| returns `nil`                          | `stopped`, and not restarted: the work is done                          |
| returns an error, or panics            | `restarting`, then run again after a backoff, see [Restarts](#restarts) |
| returns after its context was canceled | `stopped`: the application is stopping                                  |

A panic is recovered, logged with its stack, and handled like an error wrapping `worker.ErrPanic`.

## Replicas

`worker.WithReplicas(n)` runs `n` concurrent replicas of the worker, each with its own `Run` call, its own restarts and its own state. Use it for queue consumers:

```go
worker.AsWorker("orders-consumer", NewOrdersConsumer, worker.WithReplicas(4))
```

## Restarts

A failed run is restarted after a backoff, starting at `1s` and doubling up to `30s`. After `5` consecutive restarts, the replica is `failed`: it is not restarted anymore, an error is logged, the [probe](#health) fails, and the application shuts down with exit code 1, so that its orchestrator restarts it.

A run lasting at least `1m` resets the consecutive restarts: a worker failing once a day is not failed after 5 days.

Change the policy in the [configuration](/oryn/worker/configuration/#restarts), or in code with `worker.WithRestart`:

```go
worker.AsWorker("outbox", NewOutboxWorker, worker.WithRestart(worker.RestartPolicy{
  Enabled:        true,
  MaxRestarts:    0, // unlimited
  InitialBackoff: 100 * time.Millisecond,
  MaxBackoff:     time.Minute,
  ResetAfter:     time.Minute,
}))
```

The `worker.workers.<name>.*` config keys override the options set in code.

## Stop

When the application stops, the context of the workers is canceled, and the application waits for them to return, until the stop timeout of the application (`fx.StopTimeout`, 15s by default). A worker in its restart backoff is stopped right away.

## Health

The module registers a liveness and readiness probe, named `worker`, failing when a worker replica is `failed`:

```plaintext
worker test_worker (replica 0) failed: connection refused
```

A `stopped` or `disabled` worker is healthy. See [Health](/oryn/core/health/).

## Pool

The `*worker.Pool` reports the state of the workers, for example to expose them on an admin endpoint:

```go
for _, info := range pool.Workers() {
  fmt.Println(info.Name, info.Replica, info.State, info.Restarts, info.Err)
}
```

The states are `pending`, `running`, `restarting`, `stopped`, `failed` and `disabled`.

# Testing

> Run workers in tests, synchronously or with the application, and assert their telemetry.

Tests run the application with [coretest](/oryn/core/testing/), and the `workertest` package runs the workers in memory.

## Run a worker once

`workertest.Run` runs a worker once, synchronously, in the test goroutine, with the dependencies and the in-memory telemetry of the test application. The run is instrumented like with `worker.Run()`, but not restarted:

```go
func TestOutboxWorker(t *testing.T) {
  t.Parallel()


  var (
    pool  *worker.Pool
    spans *tracetest.InMemoryExporter
  )


  coretest.Start(t, core.New(worker.Module, outbox.Module), fx.Populate(&pool, &spans))


  require.NoError(t, workertest.Run(t, pool, "outbox"))
  spantest.HasSpan(t, spans, spantest.WithName("worker outbox"), spantest.WithStatus(codes.Unset))
}
```

The context of the run is the test context, canceled when the test ends: a worker looping until its context is canceled must be run with the application instead, or stop by itself once its work is done in the test, for example when its queue is empty.

## Run with the application

Add `worker.Run()` to start the workers with the test application, like in production, and wait for them to reach a state with `workertest.WaitForState`:

```go
func TestOutboxWorkerRestarts(t *testing.T) {
  t.Parallel()


  var (
    pool *worker.Pool
    logs *logtest.InMemoryExporter
  )


  coretest.Start(t, core.New(worker.Module, outbox.Module),
    config.AsOptions(config.WithValues(map[string]any{
      "worker.restart.backoff.initial": "1ms",
      "worker.shutdown_on_failure":     false,
    })),
    worker.Run(),
    fx.Populate(&pool, &logs),
  )


  infos := workertest.WaitForState(t, pool, "outbox", worker.StateFailed)
  assert.Equal(t, 5, infos[0].Restarts)
  logtest.HasRecord(t, logs, logtest.WithBody("worker failed"))
}
```

`WaitForState` waits until all the replicas of the worker are in the state, and fails the test after 5 seconds; `WaitForStateWithin` sets another timeout. The workers are stopped when the test application stops, at the end of the test.

Use a small backoff in tests, and `worker.shutdown_on_failure: false` to keep the application running when a worker is expected to fail.

The in-memory span exporter is reset when the application stops: assert the spans of a running worker before the application stops, for example once it reached the `stopped` state.

## Replace the dependencies

Replace the dependencies of a worker with fakes with `fx.Replace` or `fx.Decorate`:

```go
coretest.Start(t, core.New(worker.Module, outbox.Module),
  fx.Replace(fx.Annotate(fakePublisher, fx.As(new(outbox.Publisher)))),
  fx.Populate(&pool),
)
```