Edit pass of docs/integration-tests.md (#1198)
Co-authored-by: cperry-goog <78765543+cperry-goog@users.noreply.github.com> Co-authored-by: Chris Perry <cperry@google.com>
This commit is contained in:
starsandskies
GitHub
parent
71f1dcf39a
commit
7c4af82da4
|
|
@ -1,18 +1,18 @@
|
|||
# Integration Tests
|
||||
|
||||
This document provides a detailed overview of the integration testing framework used in this project.
|
||||
This document provides information about the integration testing framework used in this project.
|
||||
|
||||
## Overview
|
||||
|
||||
The integration tests are designed to validate the end-to-end functionality of the Gemini CLI. They execute the built binary in a controlled environment and verify that it behaves as expected when interacting with the file system.
|
||||
|
||||
These tests are located in the `integration-tests` directory and are run using a custom test runner that provides a consistent and configurable testing environment.
|
||||
These tests are located in the `integration-tests` directory and are run using a custom test runner.
|
||||
|
||||
## Running the Tests
|
||||
## Running the tests
|
||||
|
||||
The integration tests are not run as part of the default `npm run test` command. They must be run explicitly using the `npm run test:integration:sandbox:none` script.
|
||||
The integration tests are not run as part of the default `npm run test` command. They must be run explicitly using the `npm run test:integration:all` script.
|
||||
|
||||
Also as a developer for full context a shortcut can be found at
|
||||
The integration tests can also be run using the following shortcut:
|
||||
|
||||
```bash
|
||||
npm run test:e2e
|
||||
|
|
@ -20,13 +20,13 @@ npm run test:e2e
|
|||
|
||||
## Running a specfic set of tests
|
||||
|
||||
To run a 1 or more test files you can use `npm run <integration test command> <file_name1> ....` where <integration test command> is any of `test:e2e` or `test:integration*` and <file_name> is any of the files in `integration/<file_name>.test.js`
|
||||
To run a subset of test files, you can use `npm run <integration test command> <file_name1> ....` where <integration test command> is either `test:e2e` or `test:integration*` and `<file_name>` is any of the `.test.js` files in the `integration-tests/` directory. For example, the following command runs `list_directory.test.js` and `write_file.test.js`:
|
||||
|
||||
```bash
|
||||
npm run test:e2e write_file
|
||||
npm run test:e2e list_directory write_file
|
||||
```
|
||||
|
||||
### Running a Single Test by Name
|
||||
### Running a single test by name
|
||||
|
||||
To run a single test by its name, use the `--test-name-pattern` flag:
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ To run a single test by its name, use the `--test-name-pattern` flag:
|
|||
npm run test:e2e -- --test-name-pattern "reads a file"
|
||||
```
|
||||
|
||||
### Running All Tests
|
||||
### Running all tests
|
||||
|
||||
To run the entire suite of integration tests, use the following command:
|
||||
|
||||
|
|
@ -42,17 +42,13 @@ To run the entire suite of integration tests, use the following command:
|
|||
npm run test:integration:all
|
||||
```
|
||||
|
||||
### Sandbox Matrix
|
||||
### Sandbox matrix
|
||||
|
||||
The `all` command will run tests for `no sandboxing`, `docker` and `podman`.
|
||||
Each individual type can be run as
|
||||
Each individual type can be run using the following commands:
|
||||
|
||||
```bash
|
||||
npm run test:integration:all
|
||||
```
|
||||
|
||||
```bash
|
||||
npm run test:integration:sandbox-none
|
||||
npm run test:integration:sandbox:none
|
||||
```
|
||||
|
||||
```bash
|
||||
|
|
@ -67,7 +63,7 @@ npm run test:integration:sandbox:podman
|
|||
|
||||
The integration test runner provides several options for diagnostics to help track down test failures.
|
||||
|
||||
### Keeping Test Output
|
||||
### Keeping test output
|
||||
|
||||
You can preserve the temporary files created during a test run for inspection. This is useful for debugging issues with file system operations.
|
||||
|
||||
|
|
@ -83,15 +79,15 @@ KEEP_OUTPUT=true npm run test:integration:sandbox:none
|
|||
|
||||
When output is kept, the test runner will print the path to the unique directory for the test run.
|
||||
|
||||
### Verbose Output
|
||||
### Verbose output
|
||||
|
||||
For more detailed debugging, the `--verbose` flag will stream the real-time output from the `gemini` command to the console. This is useful for observing the command's behavior as it runs.
|
||||
For more detailed debugging, the `--verbose` flag streams the real-time output from the `gemini` command to the console.
|
||||
|
||||
```bash
|
||||
npm run test:integration:sandbox:none -- --verbose
|
||||
```
|
||||
|
||||
When using `--verbose` with `--keep-output`, the output is streamed to the console and also saved to a log file within the test's temporary directory.
|
||||
When using `--verbose` and `--keep-output` in the same command, the output is streamed to the console and also saved to a log file within the test's temporary directory.
|
||||
|
||||
The verbose output is formatted to clearly identify the source of the logs:
|
||||
|
||||
|
|
@ -101,11 +97,11 @@ The verbose output is formatted to clearly identify the source of the logs:
|
|||
--- END TEST: <file-name-without-js>:<test-name> ---
|
||||
```
|
||||
|
||||
## Linting and Formatting
|
||||
## Linting and formatting
|
||||
|
||||
To ensure code quality and consistency, the integration test files are linted as part of the main build process. You can also manually run the linter and auto-fixer.
|
||||
|
||||
### Running the Linter
|
||||
### Running the linter
|
||||
|
||||
To check for linting errors, run the following command:
|
||||
|
||||
|
|
@ -113,15 +109,13 @@ To check for linting errors, run the following command:
|
|||
npm run lint
|
||||
```
|
||||
|
||||
### Automatically Fixing Issues
|
||||
|
||||
To automatically fix any fixable linting errors, run:
|
||||
You can include the `--fix` flag in the command to automatically fix any fixable linting errors:
|
||||
|
||||
```bash
|
||||
npm run lint --fix
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
## Directory structure
|
||||
|
||||
The integration tests create a unique directory for each test run inside the `.integration-tests` directory. Within this directory, a subdirectory is created for each test file, and within that, a subdirectory is created for each individual test case.
|
||||
|
||||
|
|
@ -136,14 +130,12 @@ This structure makes it easy to locate the artifacts for a specific test run, fi
|
|||
└── ...other test artifacts...
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
## Continuous integration
|
||||
|
||||
To ensure the integration tests are always run, a GitHub Actions workflow is defined in `.github/workflows/e2e.yml`. This workflow automatically runs the integration tests on every pull request and push to the `main` branch.
|
||||
|
||||
The workflow uses a matrix strategy to run the tests in different sandboxing environments:
|
||||
The workflow runs the tests in different sandboxing environments to ensure Gemini CLI is tested across each:
|
||||
|
||||
- `sandbox:none`: Runs the tests without any sandboxing.
|
||||
- `sandbox:docker`: Runs the tests in a Docker container.
|
||||
- `sandbox:podman`: Runs the tests in a Podman container.
|
||||
|
||||
This ensures that the Gemini CLI is tested across a variety of environments, improving its robustness and reliability.
|
||||
|
|
|
|||
Loading…
Reference in New Issue