[POC] Manual integration release and signing

[dkirov-dd]

Summary

Implements a proof-of-concept for manually uploading integration wheels to S3 with TUF (The Update Framework) signing support, enabling secure distribution with cryptographic verification.

Key Features

• Enhanced Build Command: Generates pointer files with complete metadata (name, version, URI, digest, length, custom fields)
• Smart Upload Command:
  • Organizes files in S3 with proper structure (simple/, pointers/, metadata/)
  • Generates PEP 503 compliant indexes using constant-complexity approach (no wheel downloads)
  • Idempotent uploads skip re-uploading identical versions
• New Sign Command: Generates and signs TUF metadata (root, targets, snapshot, timestamp) with Ed25519 keys
• AWS-Vault Integration: Automatically handles authentication without verbose wrapper commands
• Integration Test Script: End-to-end workflow verification

Architecture

S3 Bucket Structure:

test-public-integration-wheels/
├── simple/
│   ├── index.html (root index)
│   └── datadog-{integration}/
│       ├── index.html (package index)
│       └── *.whl
├── pointers/
│   └── datadog-{integration}/
│       └── *.pointer
└── metadata/
    ├── root.json
    ├── targets.json
    ├── snapshot.json
    └── timestamp.json

Pointer File Format (YAML):

pointer:
  name: datadog-postgres
  version: 1.2.3
  uri: https://test-public-integration-wheels.s3.eu-north-1.amazonaws.com/simple/datadog-postgres/datadog_postgres-1.2.3-py3-none-any.whl
  digest: abc123...  # SHA256
  length: 12345      # Bytes
  custom: {}         # Reserved for in-toto

Usage

Before (verbose):

aws-vault exec sso-agent-integrations-dev-account-admin -- ddev release upload --public aws_neuron

After (simple):

ddev release build postgres
ddev release upload --public postgres  # Auto aws-vault!
ddev release sign --generate-keys      # Auto aws-vault!

Integration Test:

./scripts/test_tuf_workflow.sh postgres

Files Added

• datadog_checks_dev/datadog_checks/dev/tooling/simple_index.py (215 lines): PEP 503 index generation
• datadog_checks_dev/datadog_checks/dev/tooling/tuf_signing.py (393 lines): TUF metadata generation/signing
• datadog_checks_dev/datadog_checks/dev/tooling/commands/release/sign.py (119 lines): Sign command implementation
• datadog_checks_dev/datadog_checks/dev/tooling/aws_helpers.py (113 lines): AWS-vault integration helpers
• scripts/test_tuf_workflow.sh: End-to-end integration test script

Files Modified

• datadog_checks_dev/datadog_checks/dev/tooling/release.py: Enhanced pointer generation and upload logic
• datadog_checks_dev/datadog_checks/dev/tooling/commands/release/upload.py: Added aws-vault integration
• ddev/src/ddev/cli/release/__init__.py: Registered sign command

POC Limitations

• Dummy keys only: Uses auto-generated Ed25519 keys (not production-ready)
• No in-toto integration: Supply chain verification placeholder only
• Manual signing: ddev release sign must be run manually after uploads
• checks_downloader not modified: Out of scope for this POC

Future Work

• Integrate with checks_downloader to consume pointer files
• Implement production key management (HSM/KMS)
• Add in-toto supply chain verification
• Automate signing in CI/CD pipeline
• Add CDN for metadata distribution

Test Plan

• Build integration creates pointer file with all required fields
• Upload organizes files in proper S3 structure
• Upload generates PEP 503 compliant indexes without downloading wheels
• Sign command generates valid TUF metadata
• TUF metadata properly signed with Ed25519 keys
• AWS-vault integration works seamlessly
• Integration test script validates end-to-end workflow

🤖 Generated with Claude Code

[github-actions]

⚠️ Recommendation: Add qa/skip-qa label

This PR does not modify any files shipped with the agent.

To help streamline the release process, please consider adding the qa/skip-qa label if these changes do not require QA testing.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Generate pointer URIs with wrong package prefix

Pointer creation uses the folder name (package_path basename) when building the pointer URI, so it writes links like .../simple/postgres/<wheel> even though upload_package uploads wheels under simple/datadog-postgres/ via get_package_name. As a result every generated pointer file references a path that doesn’t exist in S3, so downstream consumers following the pointer/TUF metadata won’t be able to fetch the wheel.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Signing skips targets due to non-seekable S3 body

In generate_targets_metadata the pointer content is parsed directly from the S3 StreamingBody and then the code calls response['Body'].seek(0) to re-read it for hashing. StreamingBody objects are not seekable, so the seek raises and is caught by the broad exception handler, causing each pointer to be skipped and leaving targets metadata empty while the command still reports success.

Useful? React with 👍 / 👎.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.15%. Comparing base (5cf5dc7) to head (2b3e4f5).
⚠️ Report is 48 commits behind head on master.

Additional details and impacted files

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

To fix the problem, replace the substring checks in line 209 with a robust, structural check of the hostname. Specifically, parse parsed.hostname into its components, match the format [bucket_name].s3.[region].amazonaws.com, and ensure that the domain truly ends with .amazonaws.com, not that .amazonaws.com is simply present somewhere in the hostname. Use urllib.parse for URL parsing and then compare the domain suffix directly (e.g., hostname.endswith('.amazonaws.com')) and verify the presence of the S3 marker at the correct position. Consider using regular expressions to match the precise pattern required for S3 endpoints.

The code to change is in the block starting line 209:

209:             if '.s3.' in parsed.hostname and '.amazonaws.com' in parsed.hostname:
210:                 region = parsed.hostname.split('.s3.')[1].split('.amazonaws.com')[0]
211:             else:
212:                 region = None

This should be replaced by a robust check (likely using a regular expression) that matches exactly the S3 endpoint structure. For example, extract bucket, region, and ensure the domain ends with .amazonaws.com.

No new third-party dependencies are needed; only standard library modules (re) should be used if necessary.