Skip to content

test(integration): dynamic-stack CMA SDK sanity suite + AM 2.0 coverage + asset Content-Type fix (v1.10.1) - #168

Merged
reeshika-h merged 10 commits into
developmentfrom
test/cma-python-integration-rewrite
Jul 3, 2026
Merged

test(integration): dynamic-stack CMA SDK sanity suite + AM 2.0 coverage + asset Content-Type fix (v1.10.1)#168
reeshika-h merged 10 commits into
developmentfrom
test/cma-python-integration-rewrite

Conversation

@aniket-shikhare-cstk

@aniket-shikhare-cstkaniket-shikhare-cstk commented Jun 26, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds a comprehensive, dynamic-stack live integration ("sanity") test suite for the CMA Python SDK under tests/integration/, replacing the ~2-year-old API tests. Each run creates a fresh stack, exercises every SDK resource method (positive / negative / edge cases) against the live CMA API, then tears the stack down. Modeled on the JS CMA SDK sanity suite, adapted to pytest.

Note: Targets development directly. This branch also contains PR #167's asset-scanning unit tests (already incorporated here), so #167 can be closed as superseded once this merges.

What's included

Framework (tests/integration/framework/)

  • setup.py — login → create stack → management token → (optional) Personalize project → flag-gated teardown
  • capture.py — monkeypatches requests to record every call (method/url/headers/body/cURL) + transient-error retries
  • report.py — self-contained dashboard HTML report (summary, coverage-by-resource, per-test request/response/cURL); honest status (passed-with-warnings, xfail)
  • helpers.py — generators, response/error validators, tracked_assert, scan-status polling
  • conftest.py — session fixtures (ctx, stack, store, AM stack, EICAR file), pytest-order sequencing, per-test header isolation, report hooks

Coverage (tests/integration/api/, 31 files) — user, org, stack, locale, environment, asset, taxonomy, terms, extension, webhook, global field, content type, label, entry, variant group, variants, entry variants, branch, alias, role, workflow, delivery/management tokens, release, release items, bulk ops, publish queue, metadata, audit log, oauth, AM 2.0 assets. Complex content-type schemas (modular blocks, groups, references, JSON RTE) round-tripped through entries.

AM 2.0 / asset scanning — covers include_asset_scan_status=true surfacing _asset_scan_status (pending → clean | quarantined), am-prefixed UIDs in DAM-enabled orgs, and the publish-only api_version: 3.2 header. The asset-scan suite skips gracefully when the DAM-enabled org isn't configured.

SDK bug fix (v1.10.1) — included here, see commit fix(assets): …

  • Asset.update() sent a JSON body with Content-Type: multipart/form-data → API 422. Fixed to application/json.
  • Asset.replace() set a bare multipart/form-data (no boundary) → API 422. Fixed to let the HTTP layer set the boundary.
  • Both verified against the live API and Contentstack's own JS SDK; version bumped 1.10.0 → 1.10.1 + CHANGELOG + unit-test assertion updated. Backward-compatible (the methods were previously always failing).

Notes for reviewers

@aniket-shikhare-cstk
test: add dynamic-stack CMA SDK integration (sanity) suite
49912d4
Add a self-contained pytest integration suite under tests/integration that
creates a fresh stack per run, exercises every SDK resource method (positive,
negative, and edge cases) against the live CMA API, and tears the stack down.
- framework/: dynamic stack setup/teardown, request+cURL capture, response/error
validators, tracked assertions, and a custom dashboard HTML report
- api/: 30 resource files with full method coverage
- data/: complex content-type schemas (modular blocks, groups, references,
JSON RTE) and entry payloads
- strict, bug-catching assertions; genuine SDK/environment issues tracked via xfail
- timestamped HTML report + cURL log written to repo root (gitignored)
Also: update AGENTS.md to document the sanity suite + env vars, add
pytest/pytest-order to requirements, gitignore secrets/reports/docs.
@aniket-shikhare-cstk
test: add asset-scanning (AM 2.0) integration coverage
a591837
Cover the asset scanning feature (DAM/AM 2.0) against both the normal org and
the AM 2.0 (DAM-enabled) org:
- normal-org scan tests (test_06_asset): upload returns _asset_scan_status
'pending'; field absent unless include_asset_scan_status=true; clean file
scans 'clean'; EICAR test file scans 'quarantined'; listing includes status
- test_31_am_assets: AM-org assets get 'am'-prefixed UIDs, full CRUD round-trip,
the same scan lifecycle, publish() with the api_version: 3.2 header (publish-
only; 404 on fetch)
- framework: am_stack fixture (stack in AM_ORG_UID; whole suite skips when
unset), runtime-generated EICAR fixture (base64-encoded so the signature is
not committed raw), wait_for_scan() polling helper, and api_version header
reset for per-test isolation
Note: the correct query param is include_asset_scan_status (the response field
is _asset_scan_status); verified live.
@aniket-shikhare-cstk
fix(assets): send correct Content-Type for update and replace (v1.10.1)
4fbb827
Asset.update() forced Content-Type: multipart/form-data while sending a JSON
body, and Asset.replace() set a bare multipart/form-data header (no boundary)
while passing files=. Both made the CMA API reject the request with 422
'Please send a valid multipart/form-data payload', and both leaked the wrong
Content-Type onto subsequent requests on the shared client.
- update(): send the JSON body as application/json (matches the JS SDK and the
live API, which returns 200)
- replace(): let the HTTP layer build the multipart body with a proper boundary
Bump version to 1.10.1 and update the asset update unit test accordingly.
@aniket-shikhare-cstk
test(integration): fix delete/bulk/branch cases + transient retries
d34dde2
- taxonomy/terms/global-field delete: drop Content-Type on the body-less DELETE
(the SDK merges application/json, which the API rejects 400/500; the JS SDK
omits it) — these now pass instead of xfail
- bulk update_workflow: provide a real target stage uid + notify field
- branch delete: poll-retry the transient 'branch not valid' (905) while the
branch finishes async provisioning
- asset update/replace: flip from xfail to passing (fixed in the SDK, v1.10.1)
- capture/setup: retry transient network errors (ReadTimeout/ConnectionError)
2x with backoff and bump the client timeout to 120s so dev11 blips don't fail
the run
@aniket-shikhare-cstk
aniket-shikhare-cstk requested a review from a team as a code ownerJune 26, 2026 12:34
@github-actions

github-actionsBot commented Jun 26, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@aniket-shikhare-cstk
aniket-shikhare-cstk changed the base branch from enhc/DX-8751 to developmentJune 26, 2026 13:01
netrajpatel
netrajpatel previously approved these changes Jun 26, 2026
View reviewed changes
CopilotAI reviewed Jun 26, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new live “sanity” integration test suite under tests/integration/ that provisions a fresh stack per run, captures all CMA traffic into an HTML/cURL report, and exercises broad SDK resource coverage. It also includes an SDK fix for asset Content-Type handling (replace() boundary + update() JSON), bumps the package version to 1.10.1, and updates the changelog/agent docs.

Changes:

  • Introduces a full dynamic-stack integration framework (setup/teardown, request capture, HTML reporting, shared store/fixtures).
  • Adds ordered, per-resource live API coverage tests (including AM 2.0 asset scanning/publish behavior).
  • Fixes asset request header behavior for replace() and update(), and updates versioning/docs.

Reviewed changes

Copilot reviewed 46 out of 51 changed files in this pull request and generated 7 comments.

Show a summary per file
FileDescription
tests/unit/assets/test_assets_unit.pyUpdates asset unit assertions (incl. new scan-status/api_version coverage).
tests/integration/pytest.iniIntegration-only pytest config (test discovery, markers, opts).
tests/integration/conftest.pySession fixtures, dynamic setup/teardown, per-test header isolation, report hooks.
tests/integration/framework/init.pyFramework package marker.
tests/integration/framework/context.pyShared run context + cross-file UID store.
tests/integration/framework/helpers.pyGenerators, retry/wait helpers, response validators, tracked assertions.
tests/integration/framework/capture.pyMonkeypatches requests.request to capture traffic + generate cURL + retry transients.
tests/integration/framework/report.pyRenders a self-contained HTML dashboard + plain cURL log.
tests/integration/framework/setup.pyLogin + dynamic stack lifecycle + management token + optional Personalize project.
tests/integration/data/init.pyData package marker.
tests/integration/data/content_types.pyContent-type schema payload factories for integration coverage.
tests/integration/data/entries.pyEntry payload factories for integration coverage.
tests/integration/data/assets/extension.htmlHTML asset used for extension upload tests.
tests/integration/api/test_01_user.pyLive tests for user endpoints and safe auth ops.
tests/integration/api/test_02_organization.pyLive tests for organization endpoints and safe ownership negatives.
tests/integration/api/test_03_stack.pyLive tests for stack endpoints (settings/users/sharing).
tests/integration/api/test_04_locale.pyLive tests for locale CRUD + fallback operations.
tests/integration/api/test_05_environment.pyLive tests for environment CRUD.
tests/integration/api/test_06_asset.pyLive tests for asset CRUD, folders, versions, publish, and scan status.
tests/integration/api/test_07_taxonomy.pyLive tests for taxonomy CRUD.
tests/integration/api/test_08_terms.pyLive tests for taxonomy terms CRUD + hierarchy/search.
tests/integration/api/test_09_extension.pyLive tests for extensions CRUD + upload.
tests/integration/api/test_10_webhook.pyLive tests for webhook CRUD + executions/logs/retry.
tests/integration/api/test_11_global_field.pyLive tests for global field CRUD + export.
tests/integration/api/test_12_content_type.pyLive tests for content type CRUD + complex schema round-trip.
tests/integration/api/test_13_label.pyLive tests for label CRUD.
tests/integration/api/test_14_entry.pyLive tests for entry CRUD, complex content, atomic ops, localization, publish.
tests/integration/api/test_15_variant_group.pyLive tests for variant group CRUD (Personalize-gated).
tests/integration/api/test_16_variants.pyLive tests for variants CRUD (Personalize-gated).
tests/integration/api/test_17_entry_variants.pyLive tests for entry variants endpoints (Personalize-gated).
tests/integration/api/test_18_branch.pyLive tests for branch CRUD with force delete + retries.
tests/integration/api/test_19_alias.pyLive tests for branch alias assign/find/fetch/delete.
tests/integration/api/test_20_role.pyLive tests for role CRUD.
tests/integration/api/test_21_workflow.pyLive tests for workflow CRUD, stages, publish rules, tasks.
tests/integration/api/test_22_delivery_token.pyLive tests for delivery token CRUD.
tests/integration/api/test_23_management_token.pyLive tests for management token CRUD.
tests/integration/api/test_24_release.pyLive tests for release CRUD + clone.
tests/integration/api/test_25_release_item.pyLive tests for release items operations.
tests/integration/api/test_26_bulk_operation.pyLive tests for bulk publish/unpublish/delete/workflow update.
tests/integration/api/test_27_publish_queue.pyLive tests for publish queue find/fetch/cancel.
tests/integration/api/test_28_metadata.pyLive tests for metadata endpoints (extension-backed).
tests/integration/api/test_29_auditlog.pyLive tests for audit log find/fetch.
tests/integration/api/test_30_oauth.pyLive tests for OAuth handler surface (authorize URL, token accessors).
tests/integration/api/test_31_am_assets.pyLive tests for AM 2.0 assets (UID prefix, scan, publish-only api_version header).
requirements.txtAdds integration-suite deps (pytest, pytest-order).
contentstack_management/assets/assets.pyFixes asset replace() multipart boundary handling and update() Content-Type logic.
contentstack_management/init.pyBumps SDK version to 1.10.1.
CHANGELOG.mdAdds v1.10.1 entry describing asset header fixes.
AGENTS.mdDocuments new integration suite usage and env vars.
.gitignoreIgnores integration outputs/env files and timestamped reports.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadtests/integration/framework/setup.py Outdated
Comment threadtests/integration/framework/setup.py
Comment threadcontentstack_management/assets/assets.py Outdated
Comment threadcontentstack_management/assets/assets.py Outdated
Comment threadtests/integration/framework/capture.py
Comment threadtests/unit/assets/test_assets_unit.py
Comment threadtests/unit/assets/test_assets_unit.py
@aniket-shikhare-cstk
fix(review): address PR#168Copilot review comments
4c64ba7
- assets.py replace(): use per-request header copy (no Content-Type) instead
of mutating shared client headers dict; wrap file open in context manager
to prevent FD leak
- assets.py update(): use per-request header copy with explicit
Content-Type: application/json, robust against prior upload/replace calls
- capture.py retry: close old FD before reopening on transient retry;
use tuple(val[2:]) to handle both 2-tuple and 3-tuple file specs without
crashing on missing index
- setup.py: remove references to non-existent .env.example in docstring
and runtime error message
- test_assets_unit.py: fix scan-status unit tests to use correct query param
name (include_asset_scan_status) and lowercase string value (true),
matching the integration suite and CMA API wire format
@github-actions

github-actionsBot commented Jun 28, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@reeshika-h
reeshika-h marked this pull request as draft June 29, 2026 04:35
reeshika-h
reeshika-h previously approved these changes Jun 29, 2026
View reviewed changes
OMpawar-21
OMpawar-21 previously approved these changes Jun 29, 2026
View reviewed changes

@OMpawar-21OMpawar-21 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aniket-shikhare-cstk
fix(assets): replace() use save/restore for Content-Type header
c0ca20c
_api_client._call_request merges self.headers into every request via
headers.update(self.headers), which overwrites any per-request copy
before the request is sent. The only effective way to suppress
Content-Type for multipart uploads is to temporarily pop it from the
shared dict and restore it in a finally block.
Also retains the with-open() fix so the file handle is always closed
after the request.
@github-actions

github-actionsBot commented Jun 29, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

reeshika-h
reeshika-h previously approved these changes Jun 29, 2026
View reviewed changes
OMpawar-21
OMpawar-21 previously approved these changes Jun 29, 2026
View reviewed changes
@reeshika-h
reeshika-h self-requested a review June 29, 2026 06:36
@aniket-shikhare-cstk
fix(workflow): set entry to workflow stage before publish_request_app…
b2eb8d1
…roval
publish_request_approval calls GET /workflow which returns 401 when the
entry has no active workflow state. wf_entry is class-scoped so
TestWorkflowPublishRules gets a fresh entry that was never assigned to
a stage (TestWorkflowStagesAndTasks sets a different instance). Fix:
call set_workflow_stage inside the test to activate the entry's workflow
state before querying. Removes the xfail marker.
@github-actions

github-actionsBot commented Jun 29, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@aniket-shikhare-cstk
fix(workflow): restore xfail on publish_request_approval with accurat…
32d4a1a
…e reason
GET /workflow (publish approval state) and POST /workflow (set stage) are
separate ACL checks. The test account has set-stage permission but not
get-approval permission on DEV11, causing 401 regardless of workflow state.
The set_workflow_stage setup step is kept — it is still correct hygiene.
Unblock: grant Workflow > Publish Approval to Owner role in the test org.
@github-actions

github-actionsBot commented Jun 29, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@github-actions

github-actionsBot commented Jun 29, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@github-actions

github-actionsBot commented Jul 3, 2026

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check TypeCount (with fixes)Without fixesThresholdResult
🔴 Critical Severity0010✅ Passed
🟠 High Severity0125✅ Passed
🟡 Medium Severity03500✅ Passed
🔵 Low Severity001000✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

SeverityBreaches (with fixes)Breaches (no fixes)SLA Threshold (with/no fixes)Status
🔴 Critical0015 / 30 days✅ Passed
🟠 High0030 / 120 days✅ Passed
🟡 Medium0090 / 365 days✅ Passed
🔵 Low00180 / 365 days✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 1
  • Medium without fixes: 3
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@reeshika-h
reeshika-h marked this pull request as ready for review July 3, 2026 07:52
@reeshika-h
reeshika-h merged commit 4bb93fb into developmentJul 3, 2026
8 checks passed
@reeshika-h
reeshika-h deleted the test/cma-python-integration-rewrite branch July 3, 2026 07:52
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@reeshika-hreeshika-hreeshika-h approved these changes
@OMpawar-21OMpawar-21OMpawar-21 approved these changes
@netrajpatelnetrajpatelAwaiting requested review from netrajpatel

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@aniket-shikhare-cstk@netrajpatel@reeshika-h@OMpawar-21