From 7c4af82da4ee4322daeb0ab78acafda4a73dc7e9 Mon Sep 17 00:00:00 2001 From: starsandskies Date: Fri, 20 Jun 2025 10:27:00 -0700 Subject: [PATCH] Edit pass of docs/integration-tests.md (#1198) Co-authored-by: cperry-goog <78765543+cperry-goog@users.noreply.github.com> Co-authored-by: Chris Perry --- docs/integration-tests.md | 52 +++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 30 deletions(-) diff --git a/docs/integration-tests.md b/docs/integration-tests.md index d86a663f..400767dc 100644 --- a/docs/integration-tests.md +++ b/docs/integration-tests.md @@ -1,18 +1,18 @@ # Integration Tests -This document provides a detailed overview of the integration testing framework used in this project. +This document provides information about the integration testing framework used in this project. ## Overview The integration tests are designed to validate the end-to-end functionality of the Gemini CLI. They execute the built binary in a controlled environment and verify that it behaves as expected when interacting with the file system. -These tests are located in the `integration-tests` directory and are run using a custom test runner that provides a consistent and configurable testing environment. +These tests are located in the `integration-tests` directory and are run using a custom test runner. -## Running the Tests +## Running the tests -The integration tests are not run as part of the default `npm run test` command. They must be run explicitly using the `npm run test:integration:sandbox:none` script. +The integration tests are not run as part of the default `npm run test` command. They must be run explicitly using the `npm run test:integration:all` script. -Also as a developer for full context a shortcut can be found at +The integration tests can also be run using the following shortcut: ```bash npm run test:e2e @@ -20,13 +20,13 @@ npm run test:e2e ## Running a specfic set of tests -To run a 1 or more test files you can use `npm run ....` where is any of `test:e2e` or `test:integration*` and is any of the files in `integration/.test.js` +To run a subset of test files, you can use `npm run ....` where is either `test:e2e` or `test:integration*` and `` is any of the `.test.js` files in the `integration-tests/` directory. For example, the following command runs `list_directory.test.js` and `write_file.test.js`: ```bash -npm run test:e2e write_file +npm run test:e2e list_directory write_file ``` -### Running a Single Test by Name +### Running a single test by name To run a single test by its name, use the `--test-name-pattern` flag: @@ -34,7 +34,7 @@ To run a single test by its name, use the `--test-name-pattern` flag: npm run test:e2e -- --test-name-pattern "reads a file" ``` -### Running All Tests +### Running all tests To run the entire suite of integration tests, use the following command: @@ -42,17 +42,13 @@ To run the entire suite of integration tests, use the following command: npm run test:integration:all ``` -### Sandbox Matrix +### Sandbox matrix The `all` command will run tests for `no sandboxing`, `docker` and `podman`. -Each individual type can be run as +Each individual type can be run using the following commands: ```bash -npm run test:integration:all -``` - -```bash -npm run test:integration:sandbox-none +npm run test:integration:sandbox:none ``` ```bash @@ -67,7 +63,7 @@ npm run test:integration:sandbox:podman The integration test runner provides several options for diagnostics to help track down test failures. -### Keeping Test Output +### Keeping test output You can preserve the temporary files created during a test run for inspection. This is useful for debugging issues with file system operations. @@ -83,15 +79,15 @@ KEEP_OUTPUT=true npm run test:integration:sandbox:none When output is kept, the test runner will print the path to the unique directory for the test run. -### Verbose Output +### Verbose output -For more detailed debugging, the `--verbose` flag will stream the real-time output from the `gemini` command to the console. This is useful for observing the command's behavior as it runs. +For more detailed debugging, the `--verbose` flag streams the real-time output from the `gemini` command to the console. ```bash npm run test:integration:sandbox:none -- --verbose ``` -When using `--verbose` with `--keep-output`, the output is streamed to the console and also saved to a log file within the test's temporary directory. +When using `--verbose` and `--keep-output` in the same command, the output is streamed to the console and also saved to a log file within the test's temporary directory. The verbose output is formatted to clearly identify the source of the logs: @@ -101,11 +97,11 @@ The verbose output is formatted to clearly identify the source of the logs: --- END TEST: : --- ``` -## Linting and Formatting +## Linting and formatting To ensure code quality and consistency, the integration test files are linted as part of the main build process. You can also manually run the linter and auto-fixer. -### Running the Linter +### Running the linter To check for linting errors, run the following command: @@ -113,15 +109,13 @@ To check for linting errors, run the following command: npm run lint ``` -### Automatically Fixing Issues - -To automatically fix any fixable linting errors, run: +You can include the `--fix` flag in the command to automatically fix any fixable linting errors: ```bash npm run lint --fix ``` -## Directory Structure +## Directory structure The integration tests create a unique directory for each test run inside the `.integration-tests` directory. Within this directory, a subdirectory is created for each test file, and within that, a subdirectory is created for each individual test case. @@ -136,14 +130,12 @@ This structure makes it easy to locate the artifacts for a specific test run, fi └── ...other test artifacts... ``` -## Continuous Integration +## Continuous integration To ensure the integration tests are always run, a GitHub Actions workflow is defined in `.github/workflows/e2e.yml`. This workflow automatically runs the integration tests on every pull request and push to the `main` branch. -The workflow uses a matrix strategy to run the tests in different sandboxing environments: +The workflow runs the tests in different sandboxing environments to ensure Gemini CLI is tested across each: - `sandbox:none`: Runs the tests without any sandboxing. - `sandbox:docker`: Runs the tests in a Docker container. - `sandbox:podman`: Runs the tests in a Podman container. - -This ensures that the Gemini CLI is tested across a variety of environments, improving its robustness and reliability.