Permission first.
Purpose always.

Start with an authorized purpose

Define why the data is needed, whose source it comes from, and what permits the intended processing. Public visibility, technical reachability, account access, and permission to reuse content are different questions. Resolve them for the actual project instead of relying on a general statement that a source is available online.

Prefer an official interface or a supported export where it serves the task. Keep the granted scope narrow and make exclusions explicit. This page is an engineering review framework, not a legal determination about a particular collection or jurisdiction.

Respect access boundaries

Do not bypass authentication, access denial, or other deliberate restrictions. Review the source’s published instructions and relevant agreements. Robots instructions are a separate coordination mechanism, not authorization to access or reuse information. The Robots Exclusion Protocol specification explicitly distinguishes the protocol from access authorization.

For social platforms, confirm which resource the application can use. Metadata, account analytics, private messages, captions, and media may involve different interfaces and permissions. An available field in one context does not establish unrestricted access in another.

Minimize personal and sensitive information

Collect fields required by the task rather than every piece of content available to the worker. Avoid building contact lists from unrelated sources or carrying full private messages into exports when a business reference would be sufficient. Separate source evidence that requires restricted retention from routine operational metadata.

Keep credentials outside public assets and exported records. Use narrowly scoped execution identities, and avoid logging authorization headers, full source documents, or unrelated personal information. Review retention and deletion before the dataset grows.

Bound the operation

Set an allowed source scope, input-size limits, concurrency limits, and a retry policy. Repeated denial or persistent failure should lead to a stop or review, not an attempt to disguise the client or overwhelm the source. A failed job with an explicit reason is preferable to uncontrolled repeated collection.

For workers that read files or fetch URLs, restrict access to the intended filesystem and network boundaries. Do not assume a container, a client application, or a model prompt enforces those boundaries automatically.

Preserve uncertainty and separate actions

A model or parser should produce candidate information under a defined contract. Keep missing, ambiguous, and rejected values visible. Do not interpret a request found in a document or email as authorization to carry out that request.

Use reviewed evidence and an appropriate approval process before consequential actions. The extraction stage should not acquire broader privileges merely because the output happens to mention a transaction, account change, or instruction.

Make review and removal practical

Trace derived records to the source identities needed for correction, revocation, or removal under the project’s process. Test whether deletion propagates to evidence stores and delivery artifacts where required. Document who reviews uncertain records and who can approve changes to the collection scope.

Continue with the source directory, email extraction guide, or self-hosting article for the relevant implementation boundary.