Object Storage Integrity and Recovery

The result you are building

Finished Result:

An object inventory and policy that proves ownership, encryption, upload validation, integrity, access isolation, version/delete recovery, lifecycle, and application behavior after regional or credential failure.

Use this guide when

• You store uploads, PDFs, images, backups, datasets, logs, or generated media.
• A deleted/overwritten object must be recoverable.
• Presigned links or cross-origin access are part of the product.

Do not use it as a substitute for

• Making a bucket public to fix one access error.
• Treating object name or extension as trustworthy content type and safe path.

Before you change anything

Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.

• Provider/account/region, bucket/container, owner, purpose, data classes, and residency.
• Object key scheme, version, checksum, size, content type, encryption, and metadata.
• IAM/resource policy, public access, presigned URL, CORS, CDN, and network path.
• Versioning, deletion protection, lifecycle, replication, inventory, and backup.
• Upload abuse, overwrite, delete, restore, stale CDN, key loss, and region failure tests.

Stop Before Proceeding:

Stop policy changes when the affected bucket/account/owner is uncertain or when broad public access would expose unrelated objects. Preserve versions and audit evidence before bulk deletion or lifecycle changes.

Understand the system before fixing it

Observe before mutating Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.

Recovery must be exercised A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.

Object keys are not filesystem paths Design keys deliberately; do not trust user filenames for identity, authorization, content type, or extraction paths.

Versioning is not a separate backup Same-account credentials or lifecycle can delete versions. Critical recovery may need independent account/region/copy and tested keys.

Evidence-to-decision map

Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.

   Evidence              Likely layer       First decisive check              What the result means

   403 on known object   Identity/policy/   Evaluate principal,               Authentication, resource policy, KMS, ownership, or key encoding
                         key                bucket/object policy,             denies access.
                                            encryption key, and exact key

   Deleted object        Version/lifecyc    List versions/delete markers      Versioning was off, version expired, or permission cannot access
   cannot restore        le                 and retention                     it.

   Old image remains     CDN/cache          Compare origin                    URL is immutable in name only or invalidation did not occur.
   after overwrite                          version/checksum and cache
                                            key

   Upload executes as    Content            Inspect magic type, headers,      Untrusted active content is served inline from a trusted origin.
   HTML                  handling           serving origin, and disposition

   Replica missing       Replication        Compare inventory, replication    Replication lag/failure or unsupported configuration breaks
   recent objects                           status, and encryption            recovery promise.
                                            permissions

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

01 Classify buckets and object flows Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: Record owner, purpose, data class, region, writers/readers, upload/download/delete paths, retention, and business recovery objective.

Read the result: Every object flow has a least-privileged identity and lifecycle.

Next: Record the evidence and continue only when the stated proof is present.

02 Design stable keys and metadata Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.

Do: Generate opaque keys, preserve sanitized original name separately, store checksum, size, detected type, owner/tenant, source, version, and retention tag.

Read the result: Object identity does not depend on attacker-controlled path or extension.

Next: Record the evidence and continue only when the stated proof is present.

03 Validate and isolate uploads Why: Inconsistent inputs create false differences and make later comparisons unreliable.

Do: Enforce size/count, magic type, malware scan, decompression limits, image/document validation, quarantine, and safe content-disposition/origin. Verify checksum after transfer.

Read the result: Unsafe or mismatched files never enter the trusted serving path.

Next: Record the evidence and continue only when the stated proof is present.

04 Enforce access and encryption Why: A decisive test reduces trial-and-error and limits unnecessary change.

Do: Block public access by default, use least-privileged IAM/resource policies, scoped short-lived presigned URLs, encryption/key recovery, CORS allowlists, and audited admin roles.

Read the result: Cross-tenant and anonymous negative tests fail; authorized paths work.

Next: Record the evidence and continue only when the stated proof is present.

Procedure continued 05 Enable version and lifecycle protection Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.

Do: Configure versioning, delete protection/retention where appropriate, lifecycle by tags/classes, incomplete-upload cleanup, inventory, and cost alerts. Review bulk policy diffs.

Read the result: Overwrite/delete can be recovered inside the declared window.

Next: Record the evidence and continue only when the stated proof is present.

06 Build independent recovery Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.

Do: Replicate or copy critical data under a distinct failure boundary, preserve encryption/key access, inventory checksums, and document region/account failover.

Read the result: Recovery copy remains available after simulated source credential loss.

Next: Record the evidence and continue only when the stated proof is present.

07 Test restore and application use Why: A result is not complete until it remains observable and repeatable after the immediate fix.

Do: Restore deleted/old versions, validate checksums/types/metadata/ACL, switch a canary application path, and verify CDN invalidation or immutable URLs.

Read the result: Recovered objects are correct, authorized, and usable by the product.

Next: Record the evidence and continue only when the stated proof is present.

Operational worksheet Evidence record Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.

• Provider/account/region, bucket/container, owner, purpose, data classes, and residency.
• Object key scheme, version, checksum, size, content type, encryption, and metadata.
• IAM/resource policy, public access, presigned URL, CORS, CDN, and network path.
• Versioning, deletion protection, lifecycle, replication, inventory, and backup.
• Upload abuse, overwrite, delete, restore, stale CDN, key loss, and region failure tests.

Acceptance scoreboard

• Bucket/account/region owner, data class, flows, retention, and recovery objectives are documented.
• Keys, metadata, checksums, types, tenant ownership, and source are stable and validated.
• Unsafe uploads remain quarantined and active content is served safely.
• Public, cross-tenant, presigned, CORS, KMS, and admin access tests pass.
• Overwrite/delete/version/lifecycle behavior matches the declared recovery window.
• Independent copy, key recovery, restore, CDN, and application canary tests pass.

Decision rule SHIP / AUTOMATE GATE Proceed only when every required acceptance check is supported by direct evidence, rollback is available, and the remaining risk is explicitly owned. Unknown is not a pass.

Minimum handoff record

• Versioned object storage integrity and recovery scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Worked example

Starting Problem:

A generated PDF is overwritten under the same key, and customers keep downloading the old CDN copy.

Evidence collected

• Origin version and checksum changed.
• CDN cache key is the stable object URL.
• No invalidation ran.
• Consumers cannot see version or generated time.

Decision Use immutable versioned object keys or explicit cache invalidation and surface the artifact version to clients.

Actions taken

• Published new PDFs under content/version keys.
• Made a pointer manifest short-lived and versioned.
• Verified checksum at upload and download.
• Added stale-cache and rollback tests.

Proof Of Completion:

Each published URL resolves to one immutable checksum; pointer changes become visible within policy; old version remains intentionally recoverable.

Why this example matters The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.

Verify, recover, and hand off

Completion tests A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.

• Bucket/account/region owner, data class, flows, retention, and recovery objectives are documented.
• Keys, metadata, checksums, types, tenant ownership, and source are stable and validated.
• Unsafe uploads remain quarantined and active content is served safely.
• Public, cross-tenant, presigned, CORS, KMS, and admin access tests pass.
• Overwrite/delete/version/lifecycle behavior matches the declared recovery window.
• Independent copy, key recovery, restore, CDN, and application canary tests pass.

Rollback or safe recovery

• Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
• Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
• Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
• Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.

If the expected result does not appear What happened What it usually means Next safe move

403 on known object Authentication, resource policy, KMS, Evaluate principal, bucket/object policy, encryption key, and exact ownership, or key encoding denies key access.

Deleted object cannot restore Versioning was off, version expired, or List versions/delete markers and retention permission cannot access it.

Old image remains after URL is immutable in name only or Compare origin version/checksum and cache key overwrite invalidation did not occur.

Upload executes as HTML Untrusted active content is served Inspect magic type, headers, serving origin, and disposition inline from a trusted origin.

Reusable handoff record

• Versioned object storage integrity and recovery scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Agent delivery contract

    Required inputs
       Field                           Type                   Requirement

       target                          object                 Versioned environment, resource, identity, or workflow being evaluated.

       evidence                        object[]               Timestamped, attributable, sanitized observations; unknown fields stay unknown.

       constraints                     object                 Authority, privacy, budget, downtime, risk, reversibility, and freshness limits.

       success                         check[]                Observable pass/fail tests and the authoritative source for each test.

    Returned output
       Field                           Type                   Requirement

       diagnosis                       object                 Likely layer, supporting and conflicting evidence, alternatives, and confidence.

       plan                            step[]                 Ordered bounded actions with owner, risk, expected proof, and stop condition.

       verification                    check[]                Observed pass/fail/unknown results, not inferred success from command exit alone.

       handoff                         object                 Sanitized evidence record, recovery state, remaining risk, and next review trigger.

    Agent refusal and escalation rules
•
      Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
•
      Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
•
      Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
•
      Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.

    Confidence rule
    Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low
    confidence; a controlled test that isolates the layer and passes verification can support high confidence.

Official reference starting points

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html
• https://cloud.google.com/storage/docs/data-protection
• https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html