feat: docs

api/AGENTS.md

@@ -0,0 +1,58 @@
+# Agent Skill Index
+
+Start with the section that best matches your need. Each entry lists the problems it solves plus key files/concepts so you know what to expect before opening it.
+
+---
+
+## Platform Foundations
+
+- **[Infrastructure Overview](agent_skills/infra.md)**  
+  When to read this:  
+  - You need to understand where a feature belongs in the architecture.  
+  - You’re wiring storage, Redis, vector stores, or OTEL.  
+  - You’re about to add CLI commands or async jobs.  
+  What it covers: configuration stack (`configs/app_config.py`, remote settings), storage entry points (`extensions/ext_storage.py`, `core/file/file_manager.py`), Redis conventions (`extensions/ext_redis.py`), plugin runtime topology, vector-store factory (`core/rag/datasource/vdb/*`), observability hooks, SSRF proxy usage, and core CLI commands.
+
+- **[Coding Style](agent_skills/coding_style.md)**  
+  When to read this:  
+  - You’re writing or reviewing backend code and need the authoritative checklist.  
+  - You’re unsure about Pydantic validators, SQLAlchemy session usage, or logging patterns.  
+  - You want the exact lint/type/test commands used in PRs.  
+  Includes: Ruff & BasedPyright commands, no-annotation policy, session examples (`with Session(db.engine, ...)`), `@field_validator` usage, logging expectations, and the rule set for file size, helpers, and package management.
+
+---
+
+## Plugin & Extension Development
+
+- **[Plugin Systems](agent_skills/plugin.md)**  
+  When to read this:  
+  - You’re building or debugging a marketplace plugin.  
+  - You need to know how manifests, providers, daemons, and migrations fit together.  
+  What it covers: plugin manifests (`core/plugin/entities/plugin.py`), installation/upgrade flows (`services/plugin/plugin_service.py`, CLI commands), runtime adapters (`core/plugin/impl/*` for tool/model/datasource/trigger/endpoint/agent), daemon coordination (`core/plugin/entities/plugin_daemon.py`), and how provider registries surface capabilities to the rest of the platform.
+
+- **[Plugin OAuth](agent_skills/plugin_oauth.md)**  
+  When to read this:  
+  - You must integrate OAuth for a plugin or datasource.  
+  - You’re handling credential encryption or refresh flows.  
+  Topics: credential storage, encryption helpers (`core/helper/provider_encryption.py`), OAuth client bootstrap (`services/plugin/oauth_service.py`, `services/plugin/plugin_parameter_service.py`), and how console/API layers expose the flows.
+
+---
+
+## Workflow Entry & Execution
+
+- **[Trigger Concepts](agent_skills/trigger.md)**  
+  When to read this:  
+  - You’re debugging why a workflow didn’t start.  
+  - You’re adding a new trigger type or hook.  
+  - You need to trace async execution, draft debugging, or webhook/schedule pipelines.  
+  Details: Start-node taxonomy, webhook & schedule internals (`core/workflow/nodes/trigger_*`, `services/trigger/*`), async orchestration (`services/async_workflow_service.py`, Celery queues), debug event bus, and storage/logging interactions.
+
+---
+
+## Additional Notes for Agents
+
+- All skill docs assume you follow the coding style guide—run Ruff/BasedPyright/tests listed there before submitting changes.  
+- When you cannot find an answer in these briefs, search the codebase using the paths referenced (e.g., `core/plugin/impl/tool.py`, `services/dataset_service.py`).  
+- If you run into cross-cutting concerns (tenancy, configuration, storage), check the infrastructure guide first; it links to most supporting modules.  
+- Keep multi-tenancy and configuration central: everything flows through `configs.dify_config` and `tenant_id`.  
+- When touching plugins or triggers, consult both the system overview and the specialised doc to ensure you adjust lifecycle, storage, and observability consistently.

api/agent_skills/coding_style.md

@@ -0,0 +1,111 @@
+## Linter
+
+- Always follow `.ruff.toml`.
+- Run `uv run ruff check --fix --unsafe-fixes`.
+- Keep each line under 100 characters (including spaces).
+
+## Code Style
+
+- `snake_case` for variables and functions.
+- `PascalCase` for classes.
+- `UPPER_CASE` for constants.
+
+## Rules
+
+- Use Pydantic v2 standard.
+- Use `uv` for package management.
+- Do not override dunder methods like `__init__`, `__iadd__`, etc.
+- Never launch services (`uv run app.py`, `flask run`, etc.); running tests under `tests/` is allowed.
+- Prefer simple functions over classes for lightweight helpers.
+- Keep files below 800 lines; split when necessary.
+- Keep code readable—no clever hacks.
+- Never use type annotations.
+- Never use `print`; log with `logger = logging.getLogger(__name__)`.
+
+## Guiding Principles
+
+- Mirror the project’s layered architecture: controller → service → core/domain.
+- Reuse existing helpers in `core/`, `services/`, and `libs/` before creating new abstractions.
+- Optimise for observability: deterministic control flow, clear logging, actionable errors.
+
+## SQLAlchemy Patterns
+
+- Models inherit from `models.base.Base`; never create ad-hoc metadata or engines.
+- Open sessions with context managers:
+
+  ```python
+  from sqlalchemy.orm import Session
+
+  with Session(db.engine, expire_on_commit=False) as session:
+      stmt = select(Workflow).where(
+          Workflow.id == workflow_id,
+          Workflow.tenant_id == tenant_id,
+      )
+      workflow = session.execute(stmt).scalar_one_or_none()
+  ```
+
+- Use SQLAlchemy expressions; avoid raw SQL unless necessary.
+- Introduce repository abstractions only for very large tables (e.g., workflow executions) to support alternative storage strategies.
+- Always scope queries by `tenant_id` and protect write paths with safeguards (`FOR UPDATE`, row counts, etc.).
+
+## Storage & External IO
+
+- Access storage via `extensions.ext_storage.storage`.
+- Use `core.helper.ssrf_proxy` for outbound HTTP fetches.
+- Background tasks that touch storage must be idempotent and log the relevant object identifiers.
+
+## Pydantic Usage
+
+- Define DTOs with Pydantic v2 models and forbid extras by default.
+- Use `@field_validator` / `@model_validator` for domain rules.
+- Example:
+
+  ```python
+  from pydantic import BaseModel, ConfigDict, HttpUrl, field_validator
+
+  class TriggerConfig(BaseModel):
+      endpoint: HttpUrl
+      secret: str
+
+      model_config = ConfigDict(extra="forbid")
+
+      @field_validator("secret")
+      def ensure_secret_prefix(cls, value: str) -> str:
+          if not value.startswith("dify_"):
+              raise ValueError("secret must start with dify_")
+          return value
+  ```
+
+## Generics & Protocols
+
+- Use `typing.Protocol` to define behavioural contracts (e.g., cache interfaces).
+- Apply generics (`TypeVar`, `Generic`) for reusable utilities like caches or providers.
+- Validate dynamic inputs at runtime when generics cannot enforce safety alone.
+
+## Error Handling & Logging
+
+- Raise domain-specific exceptions (`services/errors`, `core/errors`) and translate to HTTP responses in controllers.
+- Declare `logger = logging.getLogger(__name__)` at module top.
+- Include tenant/app/workflow identifiers in log context.
+- Log retryable events at `warning`, terminal failures at `error`.
+
+## Tooling & Checks
+
+- Format/lint: `uv run --project api --dev ruff format ./api` and `uv run --project api --dev ruff check --fix --unsafe-fixes ./api`.
+- Type checks: `uv run --directory api --dev basedpyright`.
+- Tests: `uv run --project api --dev dev/pytest/pytest_unit_tests.sh`.
+- Run all of the above before submitting your work.
+
+## Controllers & Services
+
+- Controllers: parse input via Pydantic, invoke services, return serialised responses; no business logic.
+- Services: coordinate repositories, providers, background tasks; keep side effects explicit.
+- Avoid repositories unless necessary; direct SQLAlchemy usage is preferred for typical tables.
+- Document non-obvious behaviour with concise comments.
+
+## Miscellaneous
+
+- Use `configs.dify_config` for configuration—never read environment variables directly.
+- Maintain tenant awareness end-to-end; `tenant_id` must flow through every layer touching shared resources.
+- Queue async work through `services/async_workflow_service`; implement tasks under `tasks/` with explicit queue selection.
+- Keep experimental scripts under `dev/`; do not ship them in production builds.

api/agent_skills/infra.md

@@ -0,0 +1,96 @@
+## Configuration
+
+- Import `configs.dify_config` for every runtime toggle. Do not read environment variables directly.
+- Add new settings to the proper mixin inside `configs/` (deployment, feature, middleware, etc.) so they load through `DifyConfig`.
+- Remote overrides come from the optional providers in `configs/remote_settings_sources`; keep defaults in code safe when the value is missing.
+- Example: logging pulls targets from `extensions/ext_logging.py`, and model provider URLs are assembled in `services/entities/model_provider_entities.py`.
+
+## Dependencies
+
+- Runtime dependencies live in `[project].dependencies` inside `pyproject.toml`. Optional clients go into the `storage`, `tools`, or `vdb` groups under `[dependency-groups]`.
+- Always pin versions and keep the list alphabetised. Shared tooling (lint, typing, pytest) belongs in the `dev` group.
+- When code needs a new package, explain why in the PR and run `uv lock` so the lockfile stays current.
+
+## Storage & Files
+
+- Use `extensions.ext_storage.storage` for all blob IO; it already respects the configured backend.
+- Convert files for workflows with helpers in `core/file/file_manager.py`; they handle signed URLs and multimodal payloads.
+- When writing controller logic, delegate upload quotas and metadata to `services/file_service.py` instead of touching storage directly.
+- All outbound HTTP fetches (webhooks, remote files) must go through the SSRF-safe client in `core/helper/ssrf_proxy.py`; it wraps `httpx` with the allow/deny rules configured for the platform.
+
+## Redis & Shared State
+
+- Access Redis through `extensions.ext_redis.redis_client`. For locking, reuse `redis_client.lock`.
+- Prefer higher-level helpers when available: rate limits use `libs.helper.RateLimiter`, provider metadata uses caches in `core/helper/provider_cache.py`.
+
+## Models
+
+- SQLAlchemy models sit in `models/` and inherit from the shared declarative `Base` defined in `models/base.py` (metadata configured via `models/engine.py`).
+- `models/__init__.py` exposes grouped aggregates: account/tenant models, app and conversation tables, datasets, providers, workflow runs, triggers, etc. Import from there to avoid deep path churn.
+- Follow the DDD boundary: persistence objects live in `models/`, repositories under `repositories/` translate them into domain entities, and services consume those repositories.
+- When adding a table, create the model class, register it in `models/__init__.py`, wire a repository if needed, and generate an Alembic migration as described below.
+
+## Vector Stores
+
+- Vector client implementations live in `core/rag/datasource/vdb/<provider>`, with a common factory in `core/rag/datasource/vdb/vector_factory.py` and enums in `core/rag/datasource/vdb/vector_type.py`.
+- Retrieval pipelines call these providers through `core/rag/datasource/retrieval_service.py` and dataset ingestion flows in `services/dataset_service.py`.
+- The CLI helper `flask vdb-migrate` orchestrates bulk migrations using routines in `commands.py`; reuse that pattern when adding new backend transitions.
+- To add another store, mirror the provider layout, register it with the factory, and include any schema changes in Alembic migrations.
+
+## Observability & OTEL
+
+- OpenTelemetry settings live under the observability mixin in `configs/observability`. Toggle exporters and sampling via `dify_config`, not ad-hoc env reads.
+- HTTP, Celery, Redis, SQLAlchemy, and httpx instrumentation is initialised in `extensions/ext_app_metrics.py` and `extensions/ext_request_logging.py`; reuse these hooks when adding new workers or entrypoints.
+- When creating background tasks or external calls, propagate tracing context with helpers in the existing instrumented clients (e.g. use the shared `httpx` session from `core/helper/http_client_pooling.py`).
+- If you add a new external integration, ensure spans and metrics are emitted by wiring the appropriate OTEL instrumentation package in `pyproject.toml` and configuring it in `extensions/`.
+
+## Ops Integrations
+
+- Langfuse support and other tracing bridges live under `core/ops/opik_trace`. Config toggles sit in `configs/observability`, while exporters are initialised in the OTEL extensions mentioned above.
+- External monitoring services should follow this pattern: keep client code in `core/ops`, expose switches via `dify_config`, and hook initialisation in `extensions/ext_app_metrics.py` or sibling modules.
+- Before instrumenting new code paths, check whether existing context helpers (e.g. `extensions/ext_request_logging.py`) already capture the necessary metadata.
+
+## Controllers, Services, Core
+
+- Controllers only parse HTTP input and call a service method. Keep business rules in `services/`.
+- Services enforce tenant rules, quotas, and orchestration, then call into `core/` engines (workflow execution, tools, LLMs).
+- When adding a new endpoint, search for an existing service to extend before introducing a new layer. Example: workflow APIs pipe through `services/workflow_service.py` into `core/workflow`.
+
+## Plugins, Tools, Providers
+
+- In Dify a plugin is a tenant-installable bundle that declares one or more providers (tool, model, datasource, trigger, endpoint, agent strategy) plus its resource needs and version metadata. The manifest (`core/plugin/entities/plugin.py`) mirrors what you see in the marketplace documentation.
+- Installation, upgrades, and migrations are orchestrated by `services/plugin/plugin_service.py` together with helpers such as `services/plugin/plugin_migration.py`.
+- Runtime loading happens through the implementations under `core/plugin/impl/*` (tool/model/datasource/trigger/endpoint/agent). These modules normalise plugin providers so that downstream systems (`core/tools/tool_manager.py`, `services/model_provider_service.py`, `services/trigger/*`) can treat builtin and plugin capabilities the same way.
+- For remote execution, plugin daemons (`core/plugin/entities/plugin_daemon.py`, `core/plugin/impl/plugin.py`) manage lifecycle hooks, credential forwarding, and background workers that keep plugin processes in sync with the main application.
+- Acquire tool implementations through `core/tools/tool_manager.py`; it resolves builtin, plugin, and workflow-as-tool providers uniformly, injecting the right context (tenant, credentials, runtime config).
+- To add a new plugin capability, extend the relevant `core/plugin/entities` schema and register the implementation in the matching `core/plugin/impl` module rather than importing the provider directly.
+
+## Async Workloads
+
+see `agent_skills/trigger.md` for more detailed documentation.
+
+- Enqueue background work through `services/async_workflow_service.py`. It routes jobs to the tiered Celery queues defined in `tasks/`.
+- Workers boot from `celery_entrypoint.py` and execute functions in `tasks/workflow_execution_tasks.py`, `tasks/trigger_processing_tasks.py`, etc.
+- Scheduled workflows poll from `schedule/workflow_schedule_tasks.py`. Follow the same pattern if you need new periodic jobs.
+
+## Database & Migrations
+
+- SQLAlchemy models live under `models/` and map directly to migration files in `migrations/versions`.
+- Generate migrations with `uv run --project api flask db revision --autogenerate -m "<summary>"`, then review the diff; never hand-edit the database outside Alembic.
+- Apply migrations locally using `uv run --project api flask db upgrade`; production deploys expect the same history.
+- If you add tenant-scoped data, confirm the upgrade includes tenant filters or defaults consistent with the service logic touching those tables.
+
+## CLI Commands
+
+- Maintenance commands from `commands.py` are registered on the Flask CLI. Run them via `uv run --project api flask <command>`.
+- Use the built-in `db` commands from Flask-Migrate for schema operations (`flask db upgrade`, `flask db stamp`, etc.). Only fall back to custom helpers if you need their extra behaviour.
+- Custom entries such as `flask reset-password`, `flask reset-email`, and `flask vdb-migrate` handle self-hosted account recovery and vector database migrations.
+- Before adding a new command, check whether an existing service can be reused and ensure the command guards edition-specific behaviour (many enforce `SELF_HOSTED`). Document any additions in the PR.
+- Ruff helpers are run directly with `uv`: `uv run --project api --dev ruff format ./api` for formatting and `uv run --project api --dev ruff check ./api` (add `--fix` if you want automatic fixes).
+
+## When You Add Features
+
+- Check for an existing helper or service before writing a new util.
+- Uphold tenancy: every service method should receive the tenant ID from controller wrappers such as `controllers/console/wraps.py`.
+- Update or create tests alongside behaviour changes (`tests/unit_tests` for fast coverage, `tests/integration_tests` when touching orchestrations).
+- Run `uv run --project api --dev ruff check ./api`, `uv run --directory api --dev basedpyright`, and `uv run --project api --dev dev/pytest/pytest_unit_tests.sh` before submitting changes.

api/agent_skills/plugin.md

@@ -0,0 +1 @@
+// TBD

api/agent_skills/plugin_oauth.md

@@ -0,0 +1 @@
+// TBD