dsp-job-geo-file-generation — Overview¶

Batch job of the DSP that pre-generates territorial download files (levels 2 and 3) and publishes them to SeaweedFS (S3 API). The backend serves those bytes when they exist; otherwise it uses WFS on GeoServer Download. Orchestration: dsp-core (profile object-storage). In local demo this job is usually off.

Summary¶

• Role in the DSP
• When it applies
• Flow of one run
• Coordination with migration
• Objects in the bucket
• Formats and themes
• Orphan cleanup
• Databases and configuration
• Execution in core
• Run without core
• Where to go deeper

Role in the DSP¶

Reduces load on GeoServer Download for high-volume public queries (e.g. CAR consultation scale).

When it applies¶

Flow of one run¶

flowchart LR
  dsp[(dsp-db)]
  geo[(dsp-geoserver-db)]
  job[geo-file job]
  s3[(SeaweedFS)]

  job -->|reads pending flags| dsp
  job -->|reads geom / attributes| geo
  job -->|writes CSV and GPKG| s3
  job -->|writes batch + clears flags| dsp

1. Lists territories with requires_s3_file_regeneration = true (level 2 before 3).
2. For each territory, walks themes enabled in downloadThemesConfig.json and each theme’s formats[].
3. Exports by reading dsp-geoserver-db (parity with WFS).
4. Publishes to the bucket; user-metadata generated-at (UTC ISO) — the backend uses this as lastFileGenerated.
5. Clears the flag and writes last_generated_s3_file_at only when all enabled formats for the territory were published. Partial failure keeps pending for the next run.

If the bucket does not exist, the job logs an error, does not publish, and exits without crashing the process — flags remain for the next attempt.

Coordination with migration¶

Geo-file does not read BATCH_JOB_EXECUTION_SYNC_STATE.

Columns on dsp-db (levels 2 and 3 only): requires_s3_file_regeneration, last_generated_s3_file_at. Detail: Databases — Download pendencies.

Objects in the bucket¶

Slug from name (lowercase, no accents, hyphens). Level 3 includes parent slug (homonyms). API and flags use territorial id.

The filename citizens download in the UI is not the object key — the backend builds {theme}_{level2}.{ext} in the HTTP response.

Formats and themes¶

• CSV: same layout as WFS (FID, attributes in table order, geometry as WKT).
• GeoPackage (.gpkg): generated when the theme lists gpkg in formats[]. One feature table per file, named after the theme code, geometry in the_geom (WKB). The SRID comes from the layer. When the cut has none, the geometry uses the GeoPackage undefined geographic SRS (srs_id 0); the column cannot be null. Features without geometry are skipped. The backend has no WFS fallback for this format.
• New format: implement GeoFileExporter and declare in theme formats[] — backend keys and endpoints stay the same.

Themes and codes come from downloadThemesConfig.json (generated by ./config.sh from the adopter).

Orphan cleanup¶

At the end of a run, the job lists {format}/level-2/ and {format}/level-3/ in the bucket and removes objects that do not match any valid (territory, theme, format) — for example after renaming a territory.

Databases and configuration¶

Object storage (not JDBC):

dsp:
  object-storage:
    endpoint: http://dsp-object-storage:8333
    region: us-east-1
    bucket: dsp-geo-files
    access-key: ...
    secret-key: ...
    path-style-access: true

Runtime file: config/Job-Geo-File-Generation/application/application.yaml (generated by ./config.sh from environment.object_storage in adopter-config.yaml). The bucket must exist — the job does not create it.

| Stack | Java 21, Spring Boot 3.4.2, Spring Batch, PostGIS, AWS SDK v2 (S3), Maven |

Execution in core¶

Container dsp-job-geo-file-generation: no HTTP port; starts with dsp-object-storage for a real adopter.

Run without core¶

./mvnw spring-boot:run

Requires application.yaml with three datasources, accessible bucket, and dsp-geoserver-db already populated by migration. Typical use: via core ./setup.sh.

Where to go deeper¶