Monitoring

How Monitoring Works

deco Studio includes an OpenTelemetry exporter that writes NDJSON files to the DATA_DIR directory. Data is organized into three subdirectories:

• /metrics — system and application metrics
• /logs — structured log entries for every tool invocation
• /traces — distributed traces across MCP operations

Files are org-sharded with the following path structure:

 {DATA_DIR}/{type}/{org_id}/YYYY/MM/DD/HH/{uuid}.ndjson 

A 30-day retention policy with automatic cleanup keeps disk usage in check.

Persisting Monitoring Data (S3 Sidecar)

By default, monitoring data lives on the local filesystem. To survive container restarts and enable centralized storage, set up a sidecar process that periodically syncs DATA_DIR to S3:

 # Example: sync monitoring data to S3 every 5 minutes
aws s3 sync $DATA_DIR/metrics s3://your-bucket/metrics
aws s3 sync $DATA_DIR/logs s3://your-bucket/logs
aws s3 sync $DATA_DIR/traces s3://your-bucket/traces 

Option A: ClickHouse (Recommended)

For production deployments, ClickHouse provides scalable, fast aggregations over monitoring data.

1. Set the CLICKHOUSE_URL environment variable to your ClickHouse HTTP endpoint.
2. Create tables named monitoring_logs and monitoring_metrics in your ClickHouse instance.
3. When configured, the monitoring UI queries ClickHouse directly via HTTP.

 CLICKHOUSE_URL=https://your-clickhouse-instance:8123 

Option B: DuckDB + S3

For smaller deployments that want to avoid running a separate database, you can use DuckDB with S3-mounted storage.

1. Mount your S3 bucket as a local filesystem using a tool like s3fs, goofys, or mountpoint-s3.
2. Set DATA_DIR to the mounted path.
3. deco Studio writes NDJSON files directly to the mount, and the embedded DuckDB engine reads from the same path.

 # Mount S3 bucket
mountpoint-s3 your-bucket /mnt/monitoring

# Point DATA_DIR to the mount
DATA_DIR=/mnt/monitoring 

No CLICKHOUSE_URL is needed — DuckDB queries the NDJSON files on disk.

Option C: Google Cloud Storage (OTLP via collector)

For self-hosted deployments on GCP that want no ClickHouse and no disk/sidecar, Studio can read its monitoring data directly from a GCS bucket. Studio already emits monitoring data as standard OTLP logs over the network; you point those at an OpenTelemetry Collector that writes them to GCS with the google_cloud_storage exporter, and the embedded DuckDB engine reads them back via GCS’s S3-compatible endpoint.

 Studio ──OTLP logs──▶ OTel Collector ──google_cloud_storage exporter──▶ gs://bucket/<prefix>/...
                              (OTLP JSON, native GCS client)                  ▲
                                            embedded DuckDB reads + flattens at query time 

The collector writes with a Google service account (native GCS client — no S3 signing). Studio reads via DuckDB’s httpfs , which speaks GCS’s S3-compatible API, so it needs an HMAC key. Both can use the same service account.

Starting from scratch (no bucket yet)? The steps below use the gcloud CLI — set PROJECT_ID and a globally-unique BUCKET first.

1. Create the bucket. (Pre-create it — the exporter reuses it, see step 5.)

 gcloud storage buckets create "gs://${BUCKET}" \
  --project="${PROJECT_ID}" --location=us --uniform-bucket-level-access 

2. Create a service account and grant bucket access. Bucket-scoped storage.admin covers what the exporter needs (object writes and storage.buckets.get , which reuse_if_exists calls). No project-level buckets.create is required.

 gcloud iam service-accounts create studio-monitoring --project="${PROJECT_ID}"
SA="studio-monitoring@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud storage buckets add-iam-policy-binding "gs://${BUCKET}" \
  --member="serviceAccount:${SA}" --role="roles/storage.admin" 

On GKE, bind this SA to the collector’s Kubernetes SA via Workload Identity (no key file needed). Off-GKE, create a key ( gcloud iam service-accounts keys create key.json --iam-account="${SA}" ) and mount it with GOOGLE_APPLICATION_CREDENTIALS .

3. Create an HMAC key for Studio’s read. DuckDB reads via the S3-compatible API, which needs an HMAC key on the same SA:

 gcloud storage hmac keys create "${SA}" --project="${PROJECT_ID}"
#  → accessId  = MONITORING_S3_ACCESS_KEY_ID  (GOOG1E...)
#  → secret    = MONITORING_S3_SECRET_ACCESS_KEY   (shown only once) 

4. Send Studio’s monitoring logs to your collector. Set MONITORING_OTLP_ENDPOINT (or enable the in-cluster collector) so Studio exports OTLP logs to it.

5. Configure the collector to write OTLP-JSON to GCS. Add the google_cloud_storage exporter to the collector’s logs pipeline. It marshals to OTLP JSON by default — exactly what the dashboard reads.

 processors:
  batch:
    # One object is written per flush. Keep batches bounded so each file stays
    # well under the reader's 32 MiB per-file limit, and to limit how many
    # objects each dashboard query scans. (send_batch_max_size must be >=
    # send_batch_size.)
    send_batch_size: 2048
    send_batch_max_size: 2048
    timeout: 60s
exporters:
  google_cloud_storage:
    bucket:
      name: your-bucket
      project_id: your-project   # auto-detected on GKE; required off-GCP
      region: us
      reuse_if_exists: true      # use the pre-created bucket; required for restart-safety
      file_prefix: logs
      partition:
        prefix: logs             # the read prefix (must match MONITORING_S3_PREFIX)
        format: "year=%Y/month=%m/day=%d/hour=%H"
service:
  pipelines:
    logs:
      processors: [batch]
      exporters: [google_cloud_storage] 

6. Point Studio’s reader at the same bucket:

 MONITORING_S3_BUCKET=your-bucket
MONITORING_S3_PREFIX=logs       # matches the collector's partition.prefix
MONITORING_S3_ENDPOINT=https://storage.googleapis.com
MONITORING_S3_ACCESS_KEY_ID=<hmac-key>
MONITORING_S3_SECRET_ACCESS_KEY=<hmac-secret> 

When MONITORING_S3_BUCKET is set (and CLICKHOUSE_URL is not), the dashboard reads the OTLP-JSON log files from the bucket. Metrics (calls, errors, latency percentiles) are derived from those same log rows, so there is no separate metrics store. The httpfs extension DuckDB needs is baked into the official image, so this works with strict outbound network policies.

7. Verify. Make a few tool calls through Studio, then confirm files land (the collector flushes on its batch timeout) and the dashboard populates:

 gcloud storage ls --recursive "gs://${BUCKET}/logs/" | head 

Retention (recommended). When the bucket uses the year=/month=/day= partition layout above, each dashboard query prunes the read to only the day partitions covered by the selected date range — it no longer flattens the whole prefix. Schema detection still lists the objects under the prefix, so a bucket lifecycle rule remains the practical way to bound that listing cost as history accumulates. Studio applies no retention itself — add a rule to auto-delete objects, e.g. after 30 days:

 echo '{"rule":[{"action":{"type":"Delete"},"condition":{"age":30}}]}' > /tmp/lifecycle.json
gcloud storage buckets update "gs://${BUCKET}" --lifecycle-file=/tmp/lifecycle.json 

Troubleshooting

• Collector won’t start / 409 Conflict on the bucket: set bucket.reuse_if_exists: true and pre-create the bucket (step 1).
• 403 storage.buckets.get denied on startup: the collector’s service account needs bucket-level storage.admin (or at least storage.buckets.get plus object write) — see step 2.
• Dashboard empty / “Monitoring stats unavailable”: confirm objects exist with gcloud storage ls "gs://${BUCKET}/logs/" , and confirm MONITORING_S3_PREFIX exactly matches the collector’s partition.prefix .
• Studio fails to start with a config error: when MONITORING_S3_BUCKET is set, the HMAC access key and secret are required (the DuckDB extension directory is baked into the official image).
• Malformed JSON in file … : a truncated/corrupt object under the year=/… partitions fails the whole read — DuckDB can’t skip parse errors for single-object JSON. Delete the object named in the error (a partial write from a crashed collector is the usual cause); a retention rule keeps these from accumulating. Note that leftover objects outside the year=/month=/day= layout (e.g. legacy dumps at the prefix root, or an older partition scheme) are skipped automatically — the query only reads year=*/… .
• Out of Memory Error from queryMetricTimeseries / queryLlmUsageStats : the OOM comes from flattening the whole prefix in one query. With the year=/month=/day= partition layout above, queries prune the data scan to the day partitions in the dashboard’s date range, which avoids it — so first confirm the collector writes that layout ( gcloud storage ls should show year=…/month=…/day=…/ paths) and narrow the date range. If it still OOMs on a small container, reduce parallelism with DUCKDB_THREADS (e.g. 2 ) — fewer threads lowers peak memory. Note DUCKDB_MEMORY_LIMIT defaults to ~80% of the container’s RAM and cannot exceed physical memory, so raising it past what the container has doesn’t help (and lowering it just trips the limit sooner) — give the container more memory instead. Also bound growth with a bucket retention/lifecycle rule.

Environment Variables