Skip to content

End-to-end session tracing with payloads ​

Trace a complete user journey across a browser frontend, a Vercel AI Node.js server, and a Java backend into one correlated Session with HTTP request and response payloads in Softprobe Explorer.

The Travel OTA demo (softprobe/demo-ota) keeps its application source uninstrumented on main. Its compose.blackbox.yml overlay attaches Softprobe browser recording, Vercel AI telemetry, and OpenTelemetry eBPF Instrumentation (OBI) at startup so you can run the full-stack demo on macOS or Linux in one command.

Architecture and instrumentation seams ​

A traveler chats in the browser web app (travel-web), which calls the Vercel AI Node.js server (travel-agent), which calls the Java backend (travel-ota) and an external airline NDC API:

All three tiers export telemetry using the same Ingestion Key (spk_...) and join into a single Session through gen_ai.conversation.id, session.id, and the W3C traceparent header. Each tier uses a distinct OpenTelemetry service.name (travel-web, travel-agent, travel-ota) so Explorer's Graph and Span Detail separate every service and outbound peer target.

Run the demo with Docker Compose ​

You need Docker with Compose, a Softprobe Explorer workspace, and a Google Gemini API key. Use only synthetic travel details in this demo.

1. Create an Agent in Explorer ​

In Explorer, open Agents and click + Connect agent. Name the Agent travel-ota-demo, choose Vercel AI, set the environment to demo, and copy its Ingestion Key (spk_...).

2. Clone demo-ota and start the Blackbox overlay ​

sh
git clone --depth 1 https://github.com/softprobe/demo-ota.git
cd demo-ota
export INGESTION_KEY="spk_REPLACE_WITH_INGESTION_KEY"
export GOOGLE_GENERATIVE_AI_API_KEY="YOUR_GEMINI_API_KEY"
docker compose -f compose.blackbox.yml up --build

For staging Explorer, also export:

sh
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://explorer-staging.softprobe.ai/api/thelake/v1/traces"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=staging"
export SOFTPROBE_ENVIRONMENT="staging"

3. Run a journey and inspect the Session Graph ​

Open http://127.0.0.1:4310/agent, select your configured Gemini model, and complete a synthetic search, booking, and ticket-issuance flow:

  1. Search for a flight:
    text
    Find a one-way economy flight from LHR to CDG for next Tuesday.
  2. Select an offer (or reply Prepare offer 1 for Alex Demo), then click Confirm booking when the review card appears.
  3. Click Pay ... and issue ticket with the prefilled demo card.

In Softprobe Explorer, open Sessions, filter for travel-ota-demo, and open the newest Session. Switch to Graph and click nodes to inspect Span Detail across all three tiers:

  • Web app (travel-web). RRWeb DOM replay and browser POST client spans with url.template route patterns and redacted JSON payloads.
  • Vercel AI server (travel-agent). invoke_agent root spans, chat generation spans, and execute_tool spans (travel_search, travel_prepare, travel_book, travel_order) linked by gen_ai.conversation.id.
  • Java backend (travel-ota). Inbound POST /api/flights/search, /api/flights/book, /api/flights/orders/query, and /api/flights/payandissue server spans with [REDACTED] card details, plus outbound POST /ndc/v21.3/* client spans to the external NDC airline host.

Press Ctrl+C or run docker compose -f compose.blackbox.yml down when finished.

How each tier is instrumented ​

The blackbox/ directory in demo-ota wires all three tiers without modifying the uninstrumented application source:

Browser (@softprobe/web-record) ​

blackbox/entrypoint.sh loads @softprobe/web-record/dist/browser.js via <script data-softprobe-options='...' defer> in index.html and allows the OTLP origin in Content-Security-Policy:

  • Sets serviceName: "travel-web" and exports OTLP traces and rrweb DOM recordings (maskAllInputs: true).
  • Configures httpBodyCapture.routes with wildcard * segments to trace business POST calls while skipping 1-second GET /agent/api/sessions/:id polling.
  • Captures JSON bodies up to 64 KiB and replaces cardNumber, cvv, cvc, password, and secret with [REDACTED].
  • Adopts ses_* session IDs automatically from /agent/api/sessions responses, /sessions/:id URLs, and sessionStorage.

See @softprobe/web-record and WebRecordInitOptions for every option.

Vercel AI server (@softprobe/vercel-ai-sdk) ​

blackbox/init-vercel-ai.mjs runs via NODE_OPTIONS="--import /opt/blackbox/init-vercel-ai.mjs" before src/start.mjs starts:

js
import { initVercelAiTelemetry } from "@softprobe/vercel-ai-sdk";

initVercelAiTelemetry({
  serviceName: process.env.SOFTPROBE_AI_SERVICE_NAME || "travel-agent",
});

initVercelAiTelemetry extracts W3C traceparent and ses_* session IDs from incoming browser requests, records agent, generation, and tool spans, and forwards traceparent on outgoing fetch, http, and https calls to the Java backend.

Java backend (OpenTelemetry OBI) ​

blackbox/Dockerfile.obi installs official OpenTelemetry OBI v1.0.0 in a sidecar container sharing the app's PID and network namespace. blackbox/obi.yaml selects the Java travel-ota process (OTEL_SERVICE_NAME=travel-ota), propagates W3C headers, redacts $.paymentDetails.cardNumber and $.paymentDetails.cvv on /api/flights/payandissue, and extracts JSON bodies for /api/flights/* and /ndc/v21.3/*.

See the OBI distributed traces guide and Config v2 reference for Linux kernel and container deployment details.

Use your own app ​

To apply this pattern to your own stack, assign a distinct OpenTelemetry service.name to each tier:

  1. Web app. For bundled apps (Vite, Next.js, Webpack), install @softprobe/web-record and call RecordSdk.init ({ ingestionKey, serviceName: 'my-web' }). For plain HTML pages without a bundler, add <script src="https://unpkg.com/@softprobe/web-record/dist/browser.js" data-softprobe-options='{"ingestionKey":"spk_...","serviceName":"my-web"}' defer></script> to <head> and allow your OTLP endpoint origin in connect-src. Use httpBodyCapture to scope routes or add redactJsonPaths.
  2. AI agent server. Install @softprobe/vercel-ai-sdk and @softprobe/tracing, then call initVercelAiTelemetry at server startup. For Python, OpenAI, LangChain, OpenCode, or standard OpenTelemetry GenAI (gen_ai.*), see the Python SDK (softprobe-py) and JavaScript / TypeScript SDK (softprobe-js) references.
  3. Backend services. Set OTEL_SERVICE_NAME on each backend process or Kubernetes workload and run OBI with your process selector (cmd_args_glob or open_ports) and url_path_patterns allowlist.

API references ​

Arex · Blackbox · Lisa