网页存档与取证留存

技能文档

---
name: jamditis-web-archiving
title: 网页存档与取证留存
category: 法律合规
description: 网页内容的存档、找回与证据固化，覆盖 Wayback Machine、Archive.today、Memento 等存档服务的检索级联、多存档冗余备份、ArchiveBox 自托管与法律级证据链留痕（哈希、时间戳、监管链日志）。当用户需要访问已失效网页、保存网页内容、制作网页证据存档或搭建冗余存档流程时触发。触发词：网页存档、网页找回、保存网页、证据固化、Wayback。
---

# 网页存档方法论

面向新闻、研究与法律用途的网页找回与内容留存模式：访问已不可用的页面、为引用保存快照、制作带证据链的存档。

## 技能工作流

### 步骤1：按级联顺序检索已有存档

按以下顺序尝试各服务以获得最大覆盖率：

```
┌─────────────────────────────────────────────────────────────────┐
│                    ARCHIVE RETRIEVAL CASCADE                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. Wayback Machine (archive.org)                               │
│     └─ 916B+ 页面，历史深度大，支持 API                          │
│                         ↓ 未找到                                 │
│  2. Archive.today (archive.is/archive.ph)                       │
│     └─ 按需快照                                                  │
│                         ↓ 未找到                                 │
│  3. Google Cache（可用性有限）                                   │
│     └─ 较新的页面，搜索: cache:url                               │
│                         ↓ 未找到                                 │
│  4. Bing Cache                                                  │
│     └─ 点击搜索结果的下拉箭头                                    │
│                         ↓ 未找到                                 │
│  5. Memento Time Travel（聚合器）                                │
│     └─ 同时检索多个存档库                                        │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

### 步骤2：检索与保存 Wayback Machine

检查 URL 是否已被存档：

```python
import requests
from typing import Optional
from datetime import datetime

def check_wayback_availability(url: str) -> Optional[dict]:
    """Check if URL exists in Wayback Machine."""
    api_url = f"http://archive.org/wayback/available?url={url}"

    try:
        response = requests.get(api_url, timeout=10)
        data = response.json()

        if data.get('archived_snapshots', {}).get('closest'):
            snapshot = data['archived_snapshots']['closest']
            return {
                'available': snapshot.get('available', False),
                'url': snapshot.get('url'),
                'timestamp': snapshot.get('timestamp'),
                'status': snapshot.get('status')
            }
        return None
    except Exception as e:
        return None

def get_wayback_url(url: str, timestamp: str = None) -> str:
    """Generate Wayback Machine URL for a page.

    Args:
        url: Original URL to retrieve
        timestamp: Optional YYYYMMDDHHMMSS format, or None for latest
    """
    if timestamp:
        return f"https://web.archive.org/web/{timestamp}/{url}"
    return f"https://web.archive.org/web/{url}"
```

请求 Wayback Machine 存档页面：

```python
def save_to_wayback(url: str) -> Optional[str]:
    """Request Wayback Machine to archive a URL.

    Returns the archived URL if successful.
    """
    save_url = f"https://web.archive.org/save/{url}"

    headers = {
        'User-Agent': 'Mozilla/5.0 (research-archiver)'
    }

    try:
        response = requests.get(save_url, headers=headers, timeout=60)

        # Check for successful archive
        if response.status_code == 200:
            # The archived URL is in the Content-Location header
            archived_url = response.headers.get('Content-Location')
            if archived_url:
                return f"https://web.archive.org{archived_url}"
            return response.url
        return None
    except Exception:
        return None
```

用 CDX API 查询全部历史快照：

```python
def get_all_snapshots(url: str, limit: int = 100) -> list[dict]:
    """Get all archived snapshots of a URL using CDX API.

    Returns list of snapshots with timestamps and status codes.
    """
    cdx_url = "http://web.archive.org/cdx/search/cdx"
    params = {
        'url': url,
        'output': 'json',
        'limit': limit,
        'fl': 'timestamp,original,statuscode,digest,length'
    }

    try:
        response = requests.get(cdx_url, params=params, timeout=30)
        data = response.json()

        if len(data) < 2:  # First row is headers
            return []

        headers = data[0]
        snapshots = []

        for row in data[1:]:
            snapshot = dict(zip(headers, row))
            snapshot['wayback_url'] = (
                f"https://web.archive.org/web/{snapshot['timestamp']}/{snapshot['original']}"
            )
            snapshots.append(snapshot)

        return snapshots
    except Exception:
        return []
```

### 步骤3：保存到 Archive.today

```python
import requests
from urllib.parse import quote

def save_to_archive_today(url: str) -> Optional[str]:
    """Submit URL to Archive.today for archiving.

    Note: Archive.today has rate limiting and CAPTCHA requirements.
    This function works for basic archiving but may require
    manual intervention for high-volume use.
    """
    submit_url = "https://archive.today/submit/"

    data = {
        'url': url,
        'anyway': '1'  # Archive even if recent snapshot exists
    }

    try:
        response = requests.post(submit_url, data=data, timeout=60)
        # Archive.today returns the archived URL in the response
        if response.status_code == 200:
            return response.url
        return None
    except Exception:
        return None

def search_archive_today(url: str) -> Optional[str]:
    """Search for existing Archive.today snapshot."""
    search_url = f"https://archive.today/{quote(url, safe='')}"

    try:
        response = requests.get(search_url, timeout=30, allow_redirects=True)
        if response.status_code == 200 and 'archive.today' in response.url:
            return response.url
        return None
    except Exception:
        return None
```

### 步骤4：多存档冗余备份

面向最大保存可靠性的多服务冗余：

```python
from dataclasses import dataclass
from typing import Optional, List
from concurrent.futures import ThreadPoolExecutor, as_completed

@dataclass
class ArchiveResult:
    service: str
    url: str
    archived_url: Optional[str]
    success: bool
    error: Optional[str] = None

class MultiArchiver:
    """Archive URLs to multiple services for redundancy."""

    def __init__(self):
        self.services = [
            ('wayback', self._save_wayback),
            ('archive_today', self._save_archive_today),
            ('perma_cc', self._save_perma),  # Requires API key
        ]

    def archive_url(self, url: str, parallel: bool = True) -> List[ArchiveResult]:
        """Archive URL to all services.

        Args:
            url: URL to archive
            parallel: If True, archive to all services simultaneously
        """
        results = []

        if parallel:
            with ThreadPoolExecutor(max_workers=3) as executor:
                futures = {
                    executor.submit(save_func, url): name
                    for name, save_func in self.services
                }

                for future in as_completed(futures):
                    service = futures[future]
                    try:
                        archived_url = future.result()
                        results.append(ArchiveResult(
                            service=service,
                            url=url,
                            archived_url=archived_url,
                            success=archived_url is not None
                        ))
                    except Exception as e:
                        results.append(ArchiveResult(
                            service=service,
                            url=url,
                            archived_url=None,
                            success=False,
                            error=str(e)
                        ))
        else:
            for name, save_func in self.services:
                try:
                    archived_url = save_func(url)
                    results.append(ArchiveResult(
                        service=name,
                        url=url,
                        archived_url=archived_url,
                        success=archived_url is not None
                    ))
                except Exception as e:
                    results.append(ArchiveResult(
                        service=name,
                        url=url,
                        archived_url=None,
                        success=False,
                        error=str(e)
                    ))

        return results

    def _save_wayback(self, url: str) -> Optional[str]:
        return save_to_wayback(url)

    def _save_archive_today(self, url: str) -> Optional[str]:
        return save_to_archive_today(url)

    def _save_perma(self, url: str) -> Optional[str]:
        # Requires Perma.cc API key
        # Implementation depends on having API credentials
        return None
```

### 步骤5：按用途深化

- 本地自托管存档 → [references/self-hosted-archivebox.md](references/self-hosted-archivebox.md)
- 法律证据链与 Perma.cc 固定引用 → [references/legal-evidence.md](references/legal-evidence.md)
- 浏览器书签快捷存档 → [references/quick-tools.md](references/quick-tools.md)

## 存档服务对比

| 服务 | 最适合 | API | 删除策略 | 容量上限 |
|---------|----------|-----|-----------|----------|
| **Wayback Machine** | 历史研究 | 有（免费） | 可申请删除 | 无限制 |
| **Archive.today** | 快速保存 | 无 | 永不删除 | 50MB |
| **Perma.cc** | 法律引用 | 有（免费额度） | 创建者可删 | 标准页面 |
| **ArchiveBox** | 自托管、隐私 | 本地 | 永不删除 | 磁盘空间 |
| **Conifer** | 交互式内容 | 有 | 创建者可删 | 免费 5GB |

## 错误处理与回退

```python
from enum import Enum
from typing import Optional

class ArchiveError(Enum):
    NOT_FOUND = "No archive found"
    RATE_LIMITED = "Rate limited by service"
    BLOCKED = "URL blocked from archiving"
    TIMEOUT = "Request timed out"
    SERVICE_DOWN = "Archive service unavailable"

def get_archived_page(url: str) -> tuple[Optional[str], Optional[ArchiveError]]:
    """Try all archive services with proper error handling."""

    # 1. Try Wayback Machine first
    try:
        result = check_wayback_availability(url)
        if result and result.get('available'):
            return result['url'], None
    except requests.Timeout:
        pass  # Try next service
    except Exception:
        pass

    # 2. Try Archive.today
    try:
        result = search_archive_today(url)
        if result:
            return result, None
    except Exception:
        pass

    # 3. Try Memento aggregator
    try:
        memento_url = f"http://timetravel.mementoweb.org/api/json/0/{url}"
        response = requests.get(memento_url, timeout=30)
        data = response.json()

        if data.get('mementos', {}).get('closest'):
            return data['mementos']['closest']['uri'][0], None
    except Exception:
        pass

    return None, ArchiveError.NOT_FOUND
```

## 最佳实践

### 什么时候存档

- **发布前**：把作品中引用的所有来源存档
- **突发新闻**：立即存档，内容可能变更或消失
- **法律事务**：用多个存档服务制作带时间戳的证据
- **研究**：为可复现性存档一手来源
- **社交媒体**：在帖子可能被删除前存档

### 存档冗余

始终至少存档到两个服务：

```python
def ensure_archived(url: str) -> bool:
    """Ensure URL is archived in at least 2 services."""
    archiver = MultiArchiver()
    results = archiver.archive_url(url)

    successful = [r for r in results if r.success]
    return len(successful) >= 2
```

### 限速与伦理

- 批量存档遵守 `robots.txt`
- 请求之间加延时（至少 1-3 秒）
- 未经同意不存档个人/私密页面
- 有 API key 的服务使用 API key 以获得更好配额
- 缓存结果避免重复请求

使用说明

# 网页存档与取证留存

找回已失效网页、保存网页快照并制作带证据链的存档，覆盖主流存档服务与自托管方案。

## 使用

对话中直接提出需求即可触发，例如：

- 「这个链接已经 404 了，帮我找回原始内容」
- 「把这篇报道存档到多个存档服务，防止被删」
- 「我要固定一个网页作为证据，需要哈希和时间戳」
- 「帮我在本地搭一个自动存档服务」

技能会先按检索级联（Wayback → Archive.today → Google/Bing 缓存 → Memento）查找已有存档，没有再做新存档并冗余备份到多个服务。

## 工作原理

提供 Wayback Machine 可用性检查/保存/CDX 历史快照 API、Archive.today 提交脚本与多线程多服务冗余存档器；法律用途提供证据记录（SHA-256 哈希、UTC 时间戳、监管链访问日志）与 Perma.cc API 对接；长期保存提供 ArchiveBox 自托管方案；并附服务对比表、错误回退级联与限速伦理守则。

如何安装此技能？

访问技能市场，点击「安装」按钮，按提示将技能包放入 AI 编程助手的 skills 目录即可。