From dc2481c80591e5b0058547adb9f64b122a3e47d5 Mon Sep 17 00:00:00 2001 From: Yeuoly Date: Tue, 21 Oct 2025 13:56:31 +0800 Subject: [PATCH] feat: docs --- api/AGENTS.md | 58 ++++++++++++++++ api/agent_skills/coding_style.md | 111 +++++++++++++++++++++++++++++++ api/agent_skills/infra.md | 96 ++++++++++++++++++++++++++ api/agent_skills/plugin.md | 1 + api/agent_skills/plugin_oauth.md | 1 + 5 files changed, 267 insertions(+) create mode 100644 api/AGENTS.md create mode 100644 api/agent_skills/coding_style.md create mode 100644 api/agent_skills/infra.md create mode 100644 api/agent_skills/plugin.md create mode 100644 api/agent_skills/plugin_oauth.md diff --git a/api/AGENTS.md b/api/AGENTS.md new file mode 100644 index 0000000000..6fa97cdea4 --- /dev/null +++ b/api/AGENTS.md @@ -0,0 +1,58 @@ +# Agent Skill Index + +Start with the section that best matches your need. Each entry lists the problems it solves plus key files/concepts so you know what to expect before opening it. + +--- + +## Platform Foundations + +- **[Infrastructure Overview](agent_skills/infra.md)** + When to read this: + - You need to understand where a feature belongs in the architecture. + - You’re wiring storage, Redis, vector stores, or OTEL. + - You’re about to add CLI commands or async jobs. + What it covers: configuration stack (`configs/app_config.py`, remote settings), storage entry points (`extensions/ext_storage.py`, `core/file/file_manager.py`), Redis conventions (`extensions/ext_redis.py`), plugin runtime topology, vector-store factory (`core/rag/datasource/vdb/*`), observability hooks, SSRF proxy usage, and core CLI commands. + +- **[Coding Style](agent_skills/coding_style.md)** + When to read this: + - You’re writing or reviewing backend code and need the authoritative checklist. + - You’re unsure about Pydantic validators, SQLAlchemy session usage, or logging patterns. + - You want the exact lint/type/test commands used in PRs. + Includes: Ruff & BasedPyright commands, no-annotation policy, session examples (`with Session(db.engine, ...)`), `@field_validator` usage, logging expectations, and the rule set for file size, helpers, and package management. + +--- + +## Plugin & Extension Development + +- **[Plugin Systems](agent_skills/plugin.md)** + When to read this: + - You’re building or debugging a marketplace plugin. + - You need to know how manifests, providers, daemons, and migrations fit together. + What it covers: plugin manifests (`core/plugin/entities/plugin.py`), installation/upgrade flows (`services/plugin/plugin_service.py`, CLI commands), runtime adapters (`core/plugin/impl/*` for tool/model/datasource/trigger/endpoint/agent), daemon coordination (`core/plugin/entities/plugin_daemon.py`), and how provider registries surface capabilities to the rest of the platform. + +- **[Plugin OAuth](agent_skills/plugin_oauth.md)** + When to read this: + - You must integrate OAuth for a plugin or datasource. + - You’re handling credential encryption or refresh flows. + Topics: credential storage, encryption helpers (`core/helper/provider_encryption.py`), OAuth client bootstrap (`services/plugin/oauth_service.py`, `services/plugin/plugin_parameter_service.py`), and how console/API layers expose the flows. + +--- + +## Workflow Entry & Execution + +- **[Trigger Concepts](agent_skills/trigger.md)** + When to read this: + - You’re debugging why a workflow didn’t start. + - You’re adding a new trigger type or hook. + - You need to trace async execution, draft debugging, or webhook/schedule pipelines. + Details: Start-node taxonomy, webhook & schedule internals (`core/workflow/nodes/trigger_*`, `services/trigger/*`), async orchestration (`services/async_workflow_service.py`, Celery queues), debug event bus, and storage/logging interactions. + +--- + +## Additional Notes for Agents + +- All skill docs assume you follow the coding style guide—run Ruff/BasedPyright/tests listed there before submitting changes. +- When you cannot find an answer in these briefs, search the codebase using the paths referenced (e.g., `core/plugin/impl/tool.py`, `services/dataset_service.py`). +- If you run into cross-cutting concerns (tenancy, configuration, storage), check the infrastructure guide first; it links to most supporting modules. +- Keep multi-tenancy and configuration central: everything flows through `configs.dify_config` and `tenant_id`. +- When touching plugins or triggers, consult both the system overview and the specialised doc to ensure you adjust lifecycle, storage, and observability consistently. diff --git a/api/agent_skills/coding_style.md b/api/agent_skills/coding_style.md new file mode 100644 index 0000000000..26be5990aa --- /dev/null +++ b/api/agent_skills/coding_style.md @@ -0,0 +1,111 @@ +## Linter + +- Always follow `.ruff.toml`. +- Run `uv run ruff check --fix --unsafe-fixes`. +- Keep each line under 100 characters (including spaces). + +## Code Style + +- `snake_case` for variables and functions. +- `PascalCase` for classes. +- `UPPER_CASE` for constants. + +## Rules + +- Use Pydantic v2 standard. +- Use `uv` for package management. +- Do not override dunder methods like `__init__`, `__iadd__`, etc. +- Never launch services (`uv run app.py`, `flask run`, etc.); running tests under `tests/` is allowed. +- Prefer simple functions over classes for lightweight helpers. +- Keep files below 800 lines; split when necessary. +- Keep code readable—no clever hacks. +- Never use type annotations. +- Never use `print`; log with `logger = logging.getLogger(__name__)`. + +## Guiding Principles + +- Mirror the project’s layered architecture: controller → service → core/domain. +- Reuse existing helpers in `core/`, `services/`, and `libs/` before creating new abstractions. +- Optimise for observability: deterministic control flow, clear logging, actionable errors. + +## SQLAlchemy Patterns + +- Models inherit from `models.base.Base`; never create ad-hoc metadata or engines. +- Open sessions with context managers: + + ```python + from sqlalchemy.orm import Session + + with Session(db.engine, expire_on_commit=False) as session: + stmt = select(Workflow).where( + Workflow.id == workflow_id, + Workflow.tenant_id == tenant_id, + ) + workflow = session.execute(stmt).scalar_one_or_none() + ``` + +- Use SQLAlchemy expressions; avoid raw SQL unless necessary. +- Introduce repository abstractions only for very large tables (e.g., workflow executions) to support alternative storage strategies. +- Always scope queries by `tenant_id` and protect write paths with safeguards (`FOR UPDATE`, row counts, etc.). + +## Storage & External IO + +- Access storage via `extensions.ext_storage.storage`. +- Use `core.helper.ssrf_proxy` for outbound HTTP fetches. +- Background tasks that touch storage must be idempotent and log the relevant object identifiers. + +## Pydantic Usage + +- Define DTOs with Pydantic v2 models and forbid extras by default. +- Use `@field_validator` / `@model_validator` for domain rules. +- Example: + + ```python + from pydantic import BaseModel, ConfigDict, HttpUrl, field_validator + + class TriggerConfig(BaseModel): + endpoint: HttpUrl + secret: str + + model_config = ConfigDict(extra="forbid") + + @field_validator("secret") + def ensure_secret_prefix(cls, value: str) -> str: + if not value.startswith("dify_"): + raise ValueError("secret must start with dify_") + return value + ``` + +## Generics & Protocols + +- Use `typing.Protocol` to define behavioural contracts (e.g., cache interfaces). +- Apply generics (`TypeVar`, `Generic`) for reusable utilities like caches or providers. +- Validate dynamic inputs at runtime when generics cannot enforce safety alone. + +## Error Handling & Logging + +- Raise domain-specific exceptions (`services/errors`, `core/errors`) and translate to HTTP responses in controllers. +- Declare `logger = logging.getLogger(__name__)` at module top. +- Include tenant/app/workflow identifiers in log context. +- Log retryable events at `warning`, terminal failures at `error`. + +## Tooling & Checks + +- Format/lint: `uv run --project api --dev ruff format ./api` and `uv run --project api --dev ruff check --fix --unsafe-fixes ./api`. +- Type checks: `uv run --directory api --dev basedpyright`. +- Tests: `uv run --project api --dev dev/pytest/pytest_unit_tests.sh`. +- Run all of the above before submitting your work. + +## Controllers & Services + +- Controllers: parse input via Pydantic, invoke services, return serialised responses; no business logic. +- Services: coordinate repositories, providers, background tasks; keep side effects explicit. +- Avoid repositories unless necessary; direct SQLAlchemy usage is preferred for typical tables. +- Document non-obvious behaviour with concise comments. + +## Miscellaneous + +- Use `configs.dify_config` for configuration—never read environment variables directly. +- Maintain tenant awareness end-to-end; `tenant_id` must flow through every layer touching shared resources. +- Queue async work through `services/async_workflow_service`; implement tasks under `tasks/` with explicit queue selection. +- Keep experimental scripts under `dev/`; do not ship them in production builds. diff --git a/api/agent_skills/infra.md b/api/agent_skills/infra.md new file mode 100644 index 0000000000..bc36c7bf64 --- /dev/null +++ b/api/agent_skills/infra.md @@ -0,0 +1,96 @@ +## Configuration + +- Import `configs.dify_config` for every runtime toggle. Do not read environment variables directly. +- Add new settings to the proper mixin inside `configs/` (deployment, feature, middleware, etc.) so they load through `DifyConfig`. +- Remote overrides come from the optional providers in `configs/remote_settings_sources`; keep defaults in code safe when the value is missing. +- Example: logging pulls targets from `extensions/ext_logging.py`, and model provider URLs are assembled in `services/entities/model_provider_entities.py`. + +## Dependencies + +- Runtime dependencies live in `[project].dependencies` inside `pyproject.toml`. Optional clients go into the `storage`, `tools`, or `vdb` groups under `[dependency-groups]`. +- Always pin versions and keep the list alphabetised. Shared tooling (lint, typing, pytest) belongs in the `dev` group. +- When code needs a new package, explain why in the PR and run `uv lock` so the lockfile stays current. + +## Storage & Files + +- Use `extensions.ext_storage.storage` for all blob IO; it already respects the configured backend. +- Convert files for workflows with helpers in `core/file/file_manager.py`; they handle signed URLs and multimodal payloads. +- When writing controller logic, delegate upload quotas and metadata to `services/file_service.py` instead of touching storage directly. +- All outbound HTTP fetches (webhooks, remote files) must go through the SSRF-safe client in `core/helper/ssrf_proxy.py`; it wraps `httpx` with the allow/deny rules configured for the platform. + +## Redis & Shared State + +- Access Redis through `extensions.ext_redis.redis_client`. For locking, reuse `redis_client.lock`. +- Prefer higher-level helpers when available: rate limits use `libs.helper.RateLimiter`, provider metadata uses caches in `core/helper/provider_cache.py`. + +## Models + +- SQLAlchemy models sit in `models/` and inherit from the shared declarative `Base` defined in `models/base.py` (metadata configured via `models/engine.py`). +- `models/__init__.py` exposes grouped aggregates: account/tenant models, app and conversation tables, datasets, providers, workflow runs, triggers, etc. Import from there to avoid deep path churn. +- Follow the DDD boundary: persistence objects live in `models/`, repositories under `repositories/` translate them into domain entities, and services consume those repositories. +- When adding a table, create the model class, register it in `models/__init__.py`, wire a repository if needed, and generate an Alembic migration as described below. + +## Vector Stores + +- Vector client implementations live in `core/rag/datasource/vdb/`, with a common factory in `core/rag/datasource/vdb/vector_factory.py` and enums in `core/rag/datasource/vdb/vector_type.py`. +- Retrieval pipelines call these providers through `core/rag/datasource/retrieval_service.py` and dataset ingestion flows in `services/dataset_service.py`. +- The CLI helper `flask vdb-migrate` orchestrates bulk migrations using routines in `commands.py`; reuse that pattern when adding new backend transitions. +- To add another store, mirror the provider layout, register it with the factory, and include any schema changes in Alembic migrations. + +## Observability & OTEL + +- OpenTelemetry settings live under the observability mixin in `configs/observability`. Toggle exporters and sampling via `dify_config`, not ad-hoc env reads. +- HTTP, Celery, Redis, SQLAlchemy, and httpx instrumentation is initialised in `extensions/ext_app_metrics.py` and `extensions/ext_request_logging.py`; reuse these hooks when adding new workers or entrypoints. +- When creating background tasks or external calls, propagate tracing context with helpers in the existing instrumented clients (e.g. use the shared `httpx` session from `core/helper/http_client_pooling.py`). +- If you add a new external integration, ensure spans and metrics are emitted by wiring the appropriate OTEL instrumentation package in `pyproject.toml` and configuring it in `extensions/`. + +## Ops Integrations + +- Langfuse support and other tracing bridges live under `core/ops/opik_trace`. Config toggles sit in `configs/observability`, while exporters are initialised in the OTEL extensions mentioned above. +- External monitoring services should follow this pattern: keep client code in `core/ops`, expose switches via `dify_config`, and hook initialisation in `extensions/ext_app_metrics.py` or sibling modules. +- Before instrumenting new code paths, check whether existing context helpers (e.g. `extensions/ext_request_logging.py`) already capture the necessary metadata. + +## Controllers, Services, Core + +- Controllers only parse HTTP input and call a service method. Keep business rules in `services/`. +- Services enforce tenant rules, quotas, and orchestration, then call into `core/` engines (workflow execution, tools, LLMs). +- When adding a new endpoint, search for an existing service to extend before introducing a new layer. Example: workflow APIs pipe through `services/workflow_service.py` into `core/workflow`. + +## Plugins, Tools, Providers + +- In Dify a plugin is a tenant-installable bundle that declares one or more providers (tool, model, datasource, trigger, endpoint, agent strategy) plus its resource needs and version metadata. The manifest (`core/plugin/entities/plugin.py`) mirrors what you see in the marketplace documentation. +- Installation, upgrades, and migrations are orchestrated by `services/plugin/plugin_service.py` together with helpers such as `services/plugin/plugin_migration.py`. +- Runtime loading happens through the implementations under `core/plugin/impl/*` (tool/model/datasource/trigger/endpoint/agent). These modules normalise plugin providers so that downstream systems (`core/tools/tool_manager.py`, `services/model_provider_service.py`, `services/trigger/*`) can treat builtin and plugin capabilities the same way. +- For remote execution, plugin daemons (`core/plugin/entities/plugin_daemon.py`, `core/plugin/impl/plugin.py`) manage lifecycle hooks, credential forwarding, and background workers that keep plugin processes in sync with the main application. +- Acquire tool implementations through `core/tools/tool_manager.py`; it resolves builtin, plugin, and workflow-as-tool providers uniformly, injecting the right context (tenant, credentials, runtime config). +- To add a new plugin capability, extend the relevant `core/plugin/entities` schema and register the implementation in the matching `core/plugin/impl` module rather than importing the provider directly. + +## Async Workloads + +see `agent_skills/trigger.md` for more detailed documentation. + +- Enqueue background work through `services/async_workflow_service.py`. It routes jobs to the tiered Celery queues defined in `tasks/`. +- Workers boot from `celery_entrypoint.py` and execute functions in `tasks/workflow_execution_tasks.py`, `tasks/trigger_processing_tasks.py`, etc. +- Scheduled workflows poll from `schedule/workflow_schedule_tasks.py`. Follow the same pattern if you need new periodic jobs. + +## Database & Migrations + +- SQLAlchemy models live under `models/` and map directly to migration files in `migrations/versions`. +- Generate migrations with `uv run --project api flask db revision --autogenerate -m ""`, then review the diff; never hand-edit the database outside Alembic. +- Apply migrations locally using `uv run --project api flask db upgrade`; production deploys expect the same history. +- If you add tenant-scoped data, confirm the upgrade includes tenant filters or defaults consistent with the service logic touching those tables. + +## CLI Commands + +- Maintenance commands from `commands.py` are registered on the Flask CLI. Run them via `uv run --project api flask `. +- Use the built-in `db` commands from Flask-Migrate for schema operations (`flask db upgrade`, `flask db stamp`, etc.). Only fall back to custom helpers if you need their extra behaviour. +- Custom entries such as `flask reset-password`, `flask reset-email`, and `flask vdb-migrate` handle self-hosted account recovery and vector database migrations. +- Before adding a new command, check whether an existing service can be reused and ensure the command guards edition-specific behaviour (many enforce `SELF_HOSTED`). Document any additions in the PR. +- Ruff helpers are run directly with `uv`: `uv run --project api --dev ruff format ./api` for formatting and `uv run --project api --dev ruff check ./api` (add `--fix` if you want automatic fixes). + +## When You Add Features + +- Check for an existing helper or service before writing a new util. +- Uphold tenancy: every service method should receive the tenant ID from controller wrappers such as `controllers/console/wraps.py`. +- Update or create tests alongside behaviour changes (`tests/unit_tests` for fast coverage, `tests/integration_tests` when touching orchestrations). +- Run `uv run --project api --dev ruff check ./api`, `uv run --directory api --dev basedpyright`, and `uv run --project api --dev dev/pytest/pytest_unit_tests.sh` before submitting changes. diff --git a/api/agent_skills/plugin.md b/api/agent_skills/plugin.md new file mode 100644 index 0000000000..954ddd236b --- /dev/null +++ b/api/agent_skills/plugin.md @@ -0,0 +1 @@ +// TBD diff --git a/api/agent_skills/plugin_oauth.md b/api/agent_skills/plugin_oauth.md new file mode 100644 index 0000000000..954ddd236b --- /dev/null +++ b/api/agent_skills/plugin_oauth.md @@ -0,0 +1 @@ +// TBD