jcarr
/
gemini-cli
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Revert "Telemetry: Auto-export to GCP if GOOGLE_CLOUD_PROJECT is set" (#1011)

This commit is contained in:
Jerop Kipruto 2025-06-13 10:27:22 -07:00 committed by GitHub
parent daceb9963f
commit 1f63f3331f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 90 additions and 228 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

242
docs/core/telemetry.md
View File

@ -6,45 +6,6 @@ This entire system is built on the **[OpenTelemetry] (OTEL)** standard, allowing
[OpenTelemetry]: https://opentelemetry.io/
## Quick Start: Google Cloud Telemetry
This quick start guide helps you send Gemini CLI telemetry data directly to Google Cloud (Cloud Trace, Cloud Monitoring, Cloud Logging).
**Prerequisites:**
1. **Google Cloud Project:** You need an active Google Cloud Project.
2. **APIs Enabled:** Ensure the following APIs are enabled in your project:
- Cloud Trace API
- Cloud Monitoring API
- Cloud Logging API
**Steps:**
1. **Set Environment Variable:**
Set the `GOOGLE_CLOUD_PROJECT` environment variable to your project ID.
```bash
export GOOGLE_CLOUD_PROJECT="your-gcp-project-id"
```
2. **Configure Gemini CLI Settings:**
In your workspace (`.gemini/settings.json`) or user (`~/.gemini/settings.json`) settings file, enable telemetry. **Crucially, do not set `telemetryOtlpEndpoint`, or ensure it's an empty string.**
```json
{
"telemetry": true,
"sandbox": false
// "telemetryOtlpEndpoint": "" // Leave empty or omit this line
}
```
_Note: Telemetry is not compatible with sandbox mode at this time. Ensure `"sandbox": false`._
3. **Run Gemini CLI:**
That's it! The Gemini CLI will now automatically detect the `GOOGLE_CLOUD_PROJECT` variable and send telemetry data directly to your Google Cloud project.
For more detailed configuration options, including using a local collector, other OTLP backends, or advanced Google Cloud collector setups, please refer to the sections below.
## Enabling Telemetry
You can enable telemetry in multiple ways. [Configuration](configuration.md) is primarily managed via the `.gemini/settings.json` file and environment variables, but CLI flags can override these settings for a specific session.
@ -64,47 +25,13 @@ Add these lines to enable telemetry by in workspace (`.gemini/settings.json`) or
{
"telemetry": true,
"sandbox": false
// Optional: Specify a custom OTLP/gRPC endpoint for your collector.
// If commented out or empty, behavior depends on GOOGLE_CLOUD_PROJECT env var.
// "telemetryOtlpEndpoint": "http://localhost:4317"
}
```
The Gemini CLI determines where to send telemetry data based on the following priority:
## Running an OTEL Collector
1. **`telemetryOtlpEndpoint` in Settings**: If `telemetryOtlpEndpoint` is configured in `.gemini/settings.json` (and is a valid OTLP/gRPC endpoint), telemetry data will be sent to this specified endpoint. This is typically used for sending data to a local collector or a specific third-party observability platform.
_Example for a local collector:_
```json
{
"telemetry": true,
"sandbox": false,
"telemetryOtlpEndpoint": "http://localhost:4317"
}
```
2. **`GOOGLE_CLOUD_PROJECT` Environment Variable**: If `telemetryOtlpEndpoint` is not set or is empty, the CLI checks for the `GOOGLE_CLOUD_PROJECT` environment variable. If this variable is set, telemetry data (traces, metrics, and logs) will be sent directly to the corresponding Google Cloud services (Cloud Trace, Cloud Monitoring, Cloud Logging) for that project.
_To enable direct Google Cloud export, ensure `GOOGLE_CLOUD_PROJECT` is set in your environment and `telemetryOtlpEndpoint` is omitted or empty in your settings:_
```json
{
"telemetry": true,
"sandbox": false
// "telemetryOtlpEndpoint": "" // or omit the line
}
```
3. **Console Exporter (Default Fallback)**: If neither `telemetryOtlpEndpoint` is configured nor the `GOOGLE_CLOUD_PROJECT` environment variable is set, telemetry data will be exported to the console. This is useful for quick local debugging without setting up a collector or cloud services.
## Running an OTEL Collector (Optional)
While the Gemini CLI can send telemetry directly to Google Cloud (if `GOOGLE_CLOUD_PROJECT` is set) or to the console (as a fallback), you might choose to run an OpenTelemetry (OTEL) Collector in specific scenarios:
- **Local Debugging/Inspection**: To view all telemetry data locally in your terminal.
- **Custom Processing/Routing**: If you want to receive telemetry data, process it, and then forward it to one or more backends (including Google Cloud or other observability platforms).
- **Using a Non-GCP Backend**: If you want to send data to a different OTLP-compatible backend that requires a collector.
An OTEL Collector is a service that receives, processes, and exports telemetry data. When a collector is used, the CLI sends data to it using the OTLP/gRPC protocol.
An OTEL Collector is a service that receives, processes, and exports telemetry data.
The CLI sends data using the OTLP/gRPC protocol.
Learn more about OTEL exporter standard configuration in [documentation][otel-config-docs].
@ -125,11 +52,10 @@ mkdir .gemini/otel
### Local
To use a local collector, you must explicitly set the `telemetryOtlpEndpoint` in your `.gemini/settings.json` file to the collector's address (e.g., `"http://localhost:4317"`). See the "Enabling Telemetry" section for details on setting this value.
This is the simplest way to inspect events, metrics, and traces without any external tools.
This setup prints all telemetry from the Gemini CLI to your terminal using a local collector.
This setup then prints all telemetry from the Gemini CLI to your terminal using that local collector. It's the simplest way to inspect events, metrics, and traces locally without any external tools when you've configured the CLI to send data to it.
**1. Configure `telemetryOtlpEndpoint`**
**1. Create a Configuration File**
Create the file `.gemini/otel/collector-local.yaml` with the following:
@ -207,74 +133,35 @@ Use this method if you prefer not to use Docker.
### Google Cloud
The Gemini CLI can send telemetry data directly to Google Cloud services (Cloud Trace, Cloud Monitoring, Cloud Logging) if the `GOOGLE_CLOUD_PROJECT` environment variable is set and no `telemetryOtlpEndpoint` is configured in your settings. This is the simplest way to integrate with Google Cloud.
This setup sends all telemetry to Google Cloud for robust, long-term analysis.
**Direct Export to Google Cloud (Recommended for most GCP users):**
**1. Prerequisites**
1. **Prerequisites**:
- A Google Cloud Project ID.
- **APIs Enabled**: Ensure Cloud Trace API, Cloud Monitoring API, and Cloud Logging API are enabled in your Google Cloud project.
- **Authentication**: The environment where the Gemini CLI runs must be authenticated to Google Cloud with permissions to write traces, metrics, and logs. This is typically handled via Application Default Credentials (e.g., by running `gcloud auth application-default login`) or a service account with the necessary roles (`Cloud Trace Agent`, `Monitoring Metric Writer`, `Logs Writer`).
2. **Configuration**:
- A Google Cloud Project ID.
- **APIs Enabled**: Cloud Trace, Cloud Monitoring, Cloud Logging.
- **Authentication**: A Service Account with the roles `Cloud Trace Agent`, `Monitoring Metric Writer`, and `Logs Writer`. Ensure your environment is authenticated (e.g., via `gcloud auth application-default login` or a service account key file).
- Set the `GOOGLE_CLOUD_PROJECT` environment variable to your project ID.
```bash
export GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
# Optionally, set GOOGLE_CLOUD_LOCATION if needed by your setup
# export GOOGLE_CLOUD_LOCATION="YOUR_PROJECT_LOCATION" # e.g., us-central1
```
- Ensure `telemetryOtlpEndpoint` is **not set** or is empty in your `.gemini/settings.json` file.
```json
{
"telemetry": true,
"sandbox": false
// "telemetryOtlpEndpoint": "" // Ensure this is empty or commented out
}
```
**2. Set environment variables**
3. **Run Gemini CLI**: With these settings, the CLI will automatically detect `GOOGLE_CLOUD_PROJECT` and send telemetry directly to Google Cloud. No separate collector is needed for this direct export.
**Using an OTEL Collector with Google Cloud (Advanced/Custom Scenarios):**
You might choose to use an OTEL Collector if you want to:
- Perform custom processing, batching, or filtering of telemetry data before sending it to Google Cloud.
- Aggregate telemetry from multiple sources before exporting.
- Send data to Google Cloud from an environment where the CLI cannot directly authenticate or reach Google Cloud endpoints, but the collector can.
If you opt for this route, the setup involves running an OTEL Collector configured to export data to Google Cloud. The Gemini CLI would then be configured to send its telemetry to this collector's endpoint.
**1. Prerequisites (for Collector Setup)**
- All prerequisites for direct export (Project ID, APIs enabled, Authentication for the _collector_).
- An OTEL Collector setup (e.g., `otelcol-contrib` binary or Docker).
**2. Configure Gemini CLI to Send to Your Collector**
Update your `.gemini/settings.json` to point `telemetryOtlpEndpoint` to your collector's listening address (e.g., `http://localhost:4317` if the collector is local).
```json
{
"telemetry": true,
"sandbox": false,
"telemetryOtlpEndpoint": "http://localhost:4317" // Or your collector's address
}
```
**3. Create a Collector Configuration File**
Create `.gemini/otel/collector-gcp.yaml` for your collector. This file tells the collector to receive data (e.g., on `0.0.0.0:4317`) and export it to Google Cloud.
_(The existing `collector-gcp.yaml` content provided in the document can be used here, it correctly defines an OTLP receiver and a Google Cloud exporter.)_
Ensure the `project` field within the `googlecloud` exporter configuration in this YAML is correctly set, typically by referencing the `GOOGLE_CLOUD_PROJECT` environment variable available to the collector.
Set the `GOOGLE_CLOUD_PROJECT`, `GOOGLE_CLOUD_LOCATION`, and `GOOGLE_GENAI_USE_VERTEXAI` environment variables:
```bash
# Ensure GOOGLE_CLOUD_PROJECT is set in the environment where the collector runs
# export GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
GOOGLE_CLOUD_LOCATION="YOUR_PROJECT_LOCATION" # e.g., us-central1
GOOGLE_GENAI_USE_VERTEXAI=true
```
**3. Create a Configuration File**
Create `.gemini/otel/collector-gcp.yaml`:
```bash
cat <<EOF > .gemini/otel/collector-gcp.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317" # Collector listens on this endpoint
endpoint: "0.0.0.0:4317"
processors:
batch:
@ -282,12 +169,12 @@ processors:
exporters:
googlecloud:
project: "${GOOGLE_CLOUD_PROJECT}" # Collector uses this to export to GCP
project: "${GOOGLE_CLOUD_PROJECT}"
metric:
prefix: "custom.googleapis.com/gemini_cli"
log:
default_log_name: "gemini_cli"
debug: # Optional: for debugging the collector itself
debug:
verbosity: detailed
service:
@ -300,49 +187,66 @@ service:
exporters: [googlecloud]
logs:
receivers: [otlp]
exporters: [googlecloud, debug] # Sending to GCP and console for debug
exporters: [googlecloud]
EOF
```
**4. Run the Collector**
You can run the collector for Google Cloud using either Docker or a locally installed `otelcol-contrib` binary, ensuring it has access to Google Cloud credentials and the `GOOGLE_CLOUD_PROJECT` environment variable.
_(The existing Docker and `otelcol-contrib` run commands provided in the document can be used here. Ensure the collector's environment has `GOOGLE_APPLICATION_CREDENTIALS` set if using a service account key, or that it can pick up ADC.)_
**_Option 1: Use Docker_**
(Ensure `GOOGLE_CLOUD_PROJECT` is available to the Docker container if your `collector-gcp.yaml` relies on it for the `project` field, or hardcode it in the YAML.)
You can run the collector for Google Cloud using either Docker or a locally installed `otelcol` binary.
- **If using Application Default Credentials (`gcloud auth application-default login`)**:
**_Option 1: Use Docker _**
```bash
docker run --rm --name otel-collector-gcp \
-p 4317:4317 \
--user "$(id -u):$(id -g)" \
-v "$HOME/.config/gcloud/application_default_credentials.json":/etc/gcp/credentials.json:ro \
-e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/credentials.json" \
-e "GOOGLE_CLOUD_PROJECT=${GOOGLE_CLOUD_PROJECT}" \ # Pass the env var
-v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
```
This method encapsulates the collector and its dependencies within a container.
- **If using a Service Account Key File**:
```bash
docker run --rm --name otel-collector-gcp \
-p 4317:4317 \
-v "/path/to/your/sa-key.json":/etc/gcp/sa-key.json:ro \
-e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/sa-key.json" \
-e "GOOGLE_CLOUD_PROJECT=${GOOGLE_CLOUD_PROJECT}" \ # Pass the env var
-v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
```
1. **Run the Collector**:
Choose the command that matches your authentication method.
- **If using Application Default Credentials (`gcloud auth application-default login`)**:
```bash
docker run --rm --name otel-collector-gcp \
-p 4317:4317 \
--user "$(id -u):$(id -g)" \
-v "$HOME/.config/gcloud/application_default_credentials.json":/etc/gcp/credentials.json:ro \
-e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/credentials.json" \
-v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
```
- **If using a Service Account Key File**:
```bash
docker run --rm --name otel-collector-gcp \
-p 4317:4317 \
-v "/path/to/your/sa-key.json":/etc/gcp/sa-key.json:ro \
-e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/sa-key.json" \
-v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
```
2. **Check Status**:
Your telemetry data will now appear in Google Cloud Trace, Monitoring, and Logging.
3. **Stop the Collector**:
```bash
docker stop otel-collector-gcp
```
**_Option 2: Use `otelcol-contrib`_**
(Ensure `GOOGLE_CLOUD_PROJECT` is set in the shell where you run this command.)
```bash
./otelcol-contrib --config="file:$(pwd)/.gemini/otel/collector-gcp.yaml"
```
Use this method if you prefer not to use Docker.
With this collector setup, the Gemini CLI sends data to your collector, and the collector then forwards it to Google Cloud.
1. **Run the Collector**:
```bash
./otelcol-contrib --config="file:$(pwd)/.gemini/otel/collector-gcp.yaml"
```
2. **Check Status**:
Your telemetry data will now appear in Google Cloud Trace, Monitoring, and Logging.
3. **Stop the Collector**:
Press `Ctrl+C` in the terminal where the collector is running.
---

3
packages/core/src/config/config.ts
View File

@ -143,7 +143,8 @@ export class Config {
this.telemetry = params.telemetry ?? false;
this.telemetryLogUserPromptsEnabled =
params.telemetryLogUserPromptsEnabled ?? true;
this.telemetryOtlpEndpoint = params.telemetryOtlpEndpoint ?? '';
this.telemetryOtlpEndpoint =
params.telemetryOtlpEndpoint ?? 'http://localhost:4317';
this.fileFilteringRespectGitIgnore =
params.fileFilteringRespectGitIgnore ?? true;
this.checkpoint = params.checkpoint ?? false;

73
packages/core/src/telemetry/sdk.ts
View File

@ -71,77 +71,34 @@ export function initializeTelemetry(config: Config): void {
'session.id': config.getSessionId(),
});
const otlpEndpointSetting = config.getTelemetryOtlpEndpoint();
const gcpProjectId = process.env.GOOGLE_CLOUD_PROJECT;
const otlpEndpoint = config.getTelemetryOtlpEndpoint();
const grpcParsedEndpoint = parseGrpcEndpoint(otlpEndpoint);
const useOtlp = !!grpcParsedEndpoint;
let spanExporter;
let logExporter;
let metricReader;
if (otlpEndpointSetting && otlpEndpointSetting.trim() !== '') {
const grpcParsedEndpoint = parseGrpcEndpoint(otlpEndpointSetting);
if (grpcParsedEndpoint) {
diag.info(`Using user-configured OTLP endpoint: ${grpcParsedEndpoint}`);
spanExporter = new OTLPTraceExporter({
const spanExporter = useOtlp
? new OTLPTraceExporter({
url: grpcParsedEndpoint,
compression: CompressionAlgorithm.GZIP,
});
logExporter = new OTLPLogExporter({
})
: new ConsoleSpanExporter();
const logExporter = useOtlp
? new OTLPLogExporter({
url: grpcParsedEndpoint,
compression: CompressionAlgorithm.GZIP,
});
metricReader = new PeriodicExportingMetricReader({
})
: new ConsoleLogRecordExporter();
const metricReader = useOtlp
? new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: grpcParsedEndpoint,
compression: CompressionAlgorithm.GZIP,
}),
exportIntervalMillis: 10000,
});
} else {
diag.warn(
`Invalid user-configured OTLP endpoint: "${otlpEndpointSetting}". Falling back to console exporter.`,
);
spanExporter = new ConsoleSpanExporter();
logExporter = new ConsoleLogRecordExporter();
metricReader = new PeriodicExportingMetricReader({
})
: new PeriodicExportingMetricReader({
exporter: new ConsoleMetricExporter(),
exportIntervalMillis: 10000,
});
}
} else if (gcpProjectId) {
diag.info(
`No OTLP endpoint configured, GOOGLE_CLOUD_PROJECT detected (${gcpProjectId}). Exporting telemetry to Google Cloud.`,
);
const gcpTraceUrl = 'https://trace.googleapis.com:443';
const gcpMetricUrl = 'https://monitoring.googleapis.com:443';
const gcpLogUrl = 'https://logging.googleapis.com:443';
spanExporter = new OTLPTraceExporter({
url: gcpTraceUrl,
compression: CompressionAlgorithm.GZIP,
});
logExporter = new OTLPLogExporter({
url: gcpLogUrl,
compression: CompressionAlgorithm.GZIP,
});
metricReader = new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: gcpMetricUrl,
compression: CompressionAlgorithm.GZIP,
}),
exportIntervalMillis: 10000,
});
} else {
diag.info(
'No OTLP endpoint or GOOGLE_CLOUD_PROJECT detected. Using console exporters.',
);
spanExporter = new ConsoleSpanExporter();
logExporter = new ConsoleLogRecordExporter();
metricReader = new PeriodicExportingMetricReader({
exporter: new ConsoleMetricExporter(),
exportIntervalMillis: 10000,
});
}
sdk = new NodeSDK({
resource,