dsp-core¶

This module is part of the DSP — see the full documentation in dsp-docs. The information below covers this module only.

Purpose¶

The dsp-core is the DSP Docker Compose orchestration hub. It does not contain application/domain code — its responsibility is to prepare and start infrastructure (databases, GeoServers, gateway, migration and geo-file jobs) and orchestrate builds of the other modules.

flowchart TD
  core["dsp-core"]
  be["dsp-backend"]
  fe["dsp-frontend"]
  job["dsp-job-data-migration"]
  geoFile["dsp-job-geo-file-generation"]
  storage["dsp-object-storage (SeaweedFS)"]
  gs["2 GeoServers + 2 Postgres/PostGIS databases"]
  gw["dsp-gateway"]

  core --> be
  core --> fe
  core --> job
  core --> geoFile
  core --> storage
  core --> gs
  core --> gw

Responsibilities¶

• Sample adopter configuration.
• Sample About page content (config/about/).
• Database initialization SQL.
• GeoServer Exhibition (map) and GeoServer Download (export WFS).
• nginx gateway (dsp-gateway) as the single entry point for the stack.
• SeaweedFS object storage (dsp-object-storage, profile object-storage) and geo-file job (dsp-job-geo-file-generation, same profile) — required for a real adopter; Brazil Demo does not start these services.
• Migration job (dsp-job-migration, profile migration).
• Three operational scripts: ./config.sh, ./setup.sh, ./start.sh.
• Automatic clone of sibling repositories when missing (with a folder-structure preview before confirmation). The script looks in the short folder (backend and the equivalents) and, if the code is not there, in the repository-named folder (dsp-*). When both are missing, the download creates the dsp-* folder. ./config.sh also clones the job if it is missing.

Prerequisites¶

The three scripts¶

./config.sh¶

Interactive wizard that writes config/adopter/adopter-config.yaml and, on reapply, generates operational files (list below). Entry: ./config.sh (no arguments).

• config/installation/installation-config.json — hierarchy, screens, KPIs, map.initialView, AOI detail panel.
• config/map/mapLayersConfig.json — WMS groups and layers, SRIDs.
• config/downloads/downloadThemesConfig.json — Downloads screen themes.
• config/about/about-config.json — About page index (when enabled); Markdown in config/about/.
• config/Job-Data-Migration/application/application.yaml — datasources and ETL mapping.
• config/Job-Geo-File-Generation/application/application.yaml — object storage (SeaweedFS); no cron in YAML.
• .env — DSP_SOURCE_*, DSP_OBJECT_STORAGE_*, and other keys the wizard fills.

These operational artifacts must not be edited manually — they are derived and regenerated on every reapply of ./config.sh. Always adjust config/adopter/adopter-config.yaml (wizard or editor) and then reapply.

Run with no arguments (./config.sh). Before the wizard, the script looks for each module in the short folder (backend, frontend, job-data-migration, job-geo-file-generation) and, if the code is not there, in the dsp-* folder. When both are missing, it offers to clone into the repository-named folder.

The wizard does not ask for batch job schedule nor write DSP_MIGRATION_CRON or DSP_GEO_FILE_GENERATION_CRON. Reapply (option 1) does not change crons in .env.

First run (no adopter-config.yaml yet): the script asks how to configure:

Later runs (when YAML already exists):

The wizard is split into 6 stages. In each, the operator answers guided questions (field and impact explained) until adopter configuration and customization are covered:

In the terminal the wizard shows 5 numbered stages (1–5) plus the optional About block at the end — in documentation, About counts as stage 6.

Object storage (SeaweedFS) is not asked in the wizard. Credentials and endpoint live in environment.object_storage in adopter-config.yaml (see adopter-config.yaml.example); ./config.sh copies this to DSP_OBJECT_STORAGE_* in .env on reapply. The geo-file job schedule (DSP_GEO_FILE_GENERATION_CRON) comes from ./setup.sh, not this stage.

./config.sh (option 2 — edit) reopens the same 6-stage wizard with current values filled in.

Jobs generated automatically¶

The wizard does not ask which fixed jobs to enable. Generated application.yaml always turns on L1, L2, L3, area of interest, and kpi-job. The layer-jobs flag becomes true when etl.layers[] has entries; otherwise false.

The area shown on KPIs does not come from the source: kpiCalculationJob computes dsp.area_of_interest.area and writes themes to dsp.kpi_measure after migration. Stage 5 sets theme_count (0–4), each theme layer, the area unit, and card colors. Detail: KPI calculation.

Advanced SQL in source_table (fixed levels)¶

In the wizard, source_table must be schema.table. For SQL subqueries (JOINs, aliases), edit adopter-config.yaml directly with a folded YAML block (>-), as in adopter-config.yaml.example. Column names below must match SELECT aliases. Then ./config.sh → Reapply.

Unified generic layers (etl.layers[])¶

Each item feeds three destinations from a single configuration:

The same source_table may appear more than once if layer_name (and thus destination table dsp.<name>) differs. When adding another layer with the same source in the wizard, structure fields (PK, FK, geometry, dates, SRID) are reused as defaults.

To not migrate or display a layer, remove it from etl.layers[] and reapply — do not use enabled: false in adopter YAML.

In stage 6/6, the adopter can decline the About page; then it stays disabled. If enabled, the wizard validates Markdown extension and file existence and generates tab ids (tab-1, tab-2, …) in about-config.json.

About page (config/about/)¶

The config/about/ folder holds sample About page content for the frontend: about-config.json.example (sample index) and demo Markdown (*.quickstart.md.example, copied in Demonstration mode). The normal wizard flow is to point to a Markdown file anywhere on the machine, which is copied to config/about/.

Adopter YAML (config/adopter/adopter-config.yaml / .yaml.example) has an about section with fields:

apply_config() generates config/about/about-config.json with automatic ids (tab-1, tab-2, …) from tab order. The backend reads these files at /config/about/ inside the image (copied at build). Variables: DSP_ABOUT_CONFIG_FILE and DSP_ABOUT_CONTENT_DIR (.env.example).

./setup.sh¶

Prepares databases, GeoServer, and (real flow) the first migration. Runs with no arguments and shows a menu with three options:

Option 2 submenu — migration plan¶

1. When should the initial load run?

2. How should migration behave afterward?

In Continuous mode, the script asks How often should the data be synchronized after the initial migration?: every day at a time; every N hours; or every N minutes (1–59, useful for local testing). The Schedule for later time is reused if the choice is “every day”.

Next (still setup step 3, option 2), it asks for the download file pre-generation cron — 5-field Unix expression for DSP_GEO_FILE_GENERATION_CRON (default: current .env value or 0 2 * * *). It should sit in a window after periodic migration when DSP_MIGRATION_CRON exists. The adopter does not type the migration expression manually; for pre-generation, enter the cron directly or accept the default with Enter.

Migration and pre-generation values are written to .env at the end of setup (step 9), along with DSP_MIGRATION_*. Re-running ./setup.sh (real adopter) repeats the questions, using .env as default.

Migration timezone: DSP_MIGRATION_TZ in .env (see .env.example). Pre-generation inherits DSP_MIGRATION_TZ when DSP_GEO_FILE_GENERATION_TZ is empty.

If config/Job-Data-Migration/application/application.yaml is still identical to the template (.example), setup.sh stops with an error — run ./config.sh or edit the file first.

Databases and layer publication¶

Before migrating, the script waits for PostgreSQL full initialization (pg_isready + dsp and data_migration schemas created by init SQL).

AOI and generic layers exist on GeoServers only after the job populated geoserver-db.

./start.sh¶

Use after ./setup.sh. It does not run migration, seed, GeoServer populate, or start databases/jobs — it only rebuilds and starts dsp-backend, dsp-frontend, and dsp-gateway (with docker compose … --no-deps, so infra is not restarted via depends_on).

Map layers and first data load remain ./setup.sh responsibility (or the scheduled job afterward). If you ran docker compose down without -v, ./start.sh does not restart databases or GeoServers — follow the command the script prints on error or run ./setup.sh again.

Databases and GeoServer¶

Only databases publish a port on the host. HTTP services are reachable only through the gateway. The two jobs do not expose HTTP — they start via Compose profile.

Dual-write flow¶

flowchart LR
  src[(Adopter JDBC source)]
  job[Migration job]
  job -->|"business + bbox/centroid"| dspdb[(dsp-db)]
  job -->|"full geometry"| exdb[(dsp-geoserver-db)]
  exdb --> gsEx[GeoServer Exhibition WMS]
  exdb --> gsDl[GeoServer Download WFS]
  exdb --> geoFile[Geo-file job]
  geoFile -->|pre-generated CSV| s3[(dsp-object-storage SeaweedFS)]
  dspdb --> be[Backend serves API]
  be --> fe[Frontend consumes API + WMS]
  be -->|WFS downloads| gsDl
  be -->|CSV S3-first| s3

Gateway (dsp-gateway)¶

nginx container that is the single entry point for the stack. Frontend, backend, and both GeoServers do not publish a port on the host — everything enters through DSP_GATEWAY_HOST_PORT (default 8026).

Configuration lives in config/Gateway/nginx/default.conf.template, is copied into the image on docker compose build (/etc/nginx/templates/) and processed by envsubst when the container starts, substituting only DSP_* variables. Volume dsp_gateway_cache holds cache only, not templates.

Both GeoServers respond at /geoserver internally, so each gets its own external prefix and a rewrite. Each also gets PROXY_BASE_URL with its public URL so GetCapabilities and web UI links are correct.

Because the gateway resolves upstreams at runtime via Docker DNS, it starts even if some service is down — it returns 502 instead of failing on boot. That enables demo mode in ./setup.sh, which does not start backend or frontend.

Cache¶

Cache is configured but off by default. It covers only WMS/WFS endpoints (/geoserver-exhibition/<workspace>/wms, /wfs and equivalents under /geoserver-download/), which have no session — GeoServer web UI and REST are excluded so admin login is not broken.

To enable, set DSP_GATEWAY_CACHE_BYPASS empty in .env and recreate the container:

# .env
DSP_GATEWAY_CACHE_BYPASS=
DSP_GATEWAY_CACHE_TTL=10m

docker compose --env-file .env up -d --force-recreate dsp-gateway

Header X-Cache-Status (HIT, MISS, BYPASS) is on every GeoServer response for checking behavior. To clear cache, remove volume dsp_gateway_cache.

Default URLs¶

Relevant environment variables (core .env)¶

.env is created automatically on first run of ./config.sh, ./setup.sh, or ./start.sh (from .env.example). You do not need to copy it manually.

./config.sh writes JDBC, SRID, and DSP_OBJECT_STORAGE_* (from environment in adopter YAML) to .env. Migration and geo-file schedules come from ./setup.sh (real adopter). Understanding the variables below helps customize the installation and troubleshoot.

See also: Full installation, Databases.

Generated configuration structure¶

config/adopter/adopter-config.yaml is the central file produced by the ./config.sh wizard. From it are derived operational files consumed by the backend (installationConfig.json, mapLayersConfig.json, downloadThemesConfig.json, about-config.json) and the migration job (application.yaml), so each module does not need isolated manual configuration.

The ./config.sh wizard produces a single adopter-config.yaml and derives all operational artifacts consumed by other modules. downloadThemesConfig.json enters this pipeline as the theme catalog for the Downloads screen and backend WFS proxy. about-config.json enters as the content index (banner + tabs) for the frontend About page.

flowchart LR
  configSh["./config.sh"]
  configSh --> installJson["installation-config.json"]
  configSh --> mapJson["mapLayersConfig.json"]
  configSh --> downloadJson["downloadThemesConfig.json"]
  configSh --> aboutJson["about-config.json"]
  configSh --> appYaml["Job-Data-Migration application.yaml"]
  configSh --> geoYaml["Job-Geo-File-Generation application.yaml"]
  configSh --> dotenvStorage[".env DSP_OBJECT_STORAGE_* / DSP_SOURCE_*"]
  installJson --> build["docker compose build<br/>dsp_config context"]
  mapJson --> build
  downloadJson --> build
  aboutJson --> build
  appYaml --> build
  geoYaml --> build
  dotenvStorage --> setupSh["./setup.sh writes crons to .env"]
  build --> backendImg["dsp-backend /config"]
  build --> geoserverImg["GeoServers /config"]
  build --> jobImg["dsp-job-migration /config"]

• ./config.sh — adopter entry point; wizard, reapply, or recreate adopter-config.yaml and generate operational JSON/YAML listed above.
• select-runtime-config.sh — on Docker build, picks active file or .example and copies to /config inside the image.
• installation-config.json — labels, hierarchy, screens, KPIs, and screens.home.detail.fields (DSP_INSTALLATION_CONFIG_FILE). That array does not go to the job application.yaml.
• mapLayersConfig.json — WMS groups and layers; published on GeoServers by populate_geoserver.sh.
• downloadThemesConfig.json — download themes (AOI + etl.layers[]); formats is csv and gpkg; wfsBaseUrl at ${DSP_PUBLIC_BASE_URL}/geoserver-download/dsp/wfs.
• about-config.json — About index (enabled, bannerTitle, tabs with ids tab-1, tab-2, …).
• application.yaml — ETL plan. Copied to job image at build (with entrypoint and GeoServer publish scripts).
• config/Job-Geo-File-Generation/application/application.yaml — S3 for pre-generation job (no cron; schedule only in .env via ./setup.sh).
• Images dsp-backend, GeoServers, dsp-job-migration, databases, and dsp-gateway — configs and init SQL copied at build via dsp_config; volumes hold data only (and gateway cache).
• Image dsp-object-storage — SeaweedFS (weed mini) with credentials in s3.json in the image. Volume dsp_object_storage_data holds objects; internal volume size and count are computed by weed mini from free space on the Docker volume.

See also: Data flow (runtime) and dsp-backend (download environment variables).