Parse Apache Logs to JSON: Combined, NGINX, Categorization, Bot Detection

1. 30-second quickstart

curl https://streamfix.dev/v1/map \
  -H "Authorization: Bearer sk_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "payload": "127.0.0.1 - alice [10/Oct/2023:13:55:36 -0700] \"GET /api/users HTTP/1.1\" 200 2326 \"https://referrer.com/\" \"Mozilla/5.0\"",
    "target": "Parse Apache combined log format. Return object with: ip, user (null if dash), timestamp_iso (ISO 8601 with timezone), method, path, status (int), bytes (int), referrer (null if dash), user_agent."
  }'

The first call compiles a parser; subsequent calls with the same log shape and target description run the cached parser in milliseconds. For high-volume ingestion, see the bulk-processing pattern in section 4.

2. Endpoint reference

POST /v1/map. The payload is the log line as a string; target is the description of the structured event you want.

3. Worked examples

127.0.0.1 - alice [10/Oct/2023:13:55:36 -0700] "GET /api/users HTTP/1.1" 200 2326 "https://referrer.com/" "Mozilla/5.0"

"Parse Apache combined log format. Return object with: ip, user (null if '-'), timestamp_iso (ISO 8601 with timezone), method, path, status (int), bytes (int), referrer (null if '-'), user_agent."

{
  "ip": "127.0.0.1",
  "user": "alice",
  "timestamp_iso": "2023-10-10T13:55:36-07:00",
  "method": "GET",
  "path": "/api/users",
  "status": 200,
  "bytes": 2326,
  "referrer": "https://referrer.com/",
  "user_agent": "Mozilla/5.0"
}

203.0.113.5 - - [22/Apr/2024:08:14:55 +0000] "POST /api/v2/orders HTTP/2.0" 201 412 "-" "curl/8.4.0" "203.0.113.5"

"Parse NGINX default log_format with x_forwarded_for. Return object with: remote_addr, time_iso (ISO 8601 with TZ), method, path, http_version, status (int), body_bytes (int), referer (null if '-'), user_agent, x_forwarded_for (null if '-')."

{
  "remote_addr": "203.0.113.5",
  "time_iso": "2024-04-22T08:14:55+00:00",
  "method": "POST",
  "path": "/api/v2/orders",
  "http_version": "2.0",
  "status": 201,
  "body_bytes": 412,
  "referer": null,
  "user_agent": "curl/8.4.0",
  "x_forwarded_for": "203.0.113.5"
}

"Parse Apache combined log. Return object with: ip, path, method, timestamp_iso, status (int), error_category ('client_error' if status is 400-499, 'server_error' if status >= 500, 'redirect' if status is 300-399, 'ok' otherwise), is_error (true if status >= 400)."

{
  "ip": "10.0.0.5",
  "path": "/missing",
  "method": "GET",
  "timestamp_iso": "2023-10-10T14:01:12-07:00",
  "status": 404,
  "error_category": "client_error",
  "is_error": true
}

"Parse Apache combined log. Return object with all standard fields (ip, timestamp_iso, method, path, status int, bytes int, user_agent) PLUS is_bot (true if user_agent contains 'bot' or 'crawl' or 'spider' case-insensitive) AND bot_name (extracted name like 'Googlebot' or null)."

{
  "ip": "66.249.66.1",
  "timestamp_iso": "2023-10-10T14:30:00-07:00",
  "method": "GET",
  "path": "/robots.txt",
  "status": 200,
  "bytes": 256,
  "user_agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
  "is_bot": true,
  "bot_name": "Googlebot"
}

"Parse multi-line Apache combined log (newline-separated). Return summary: total_requests (int), by_status (object keyed by status int, value count), top_paths (array of {path, count} sorted by count desc, top 3), total_bytes (int sum)."

{
  "total_requests": 4,
  "by_status": {"200": 2, "404": 1, "500": 1},
  "top_paths": [
    {"path": "/a", "count": 2},
    {"path": "/b", "count": 1},
    {"path": "/c", "count": 1}
  ],
  "total_bytes": 430
}

4. Bulk-processing pattern

For high-volume log ingestion, send one representative line first to compile the parser. Subsequent lines of the same shape hit the cache. Most production access logs have just one or two distinct shapes.

import httpx

TARGET = (
    "Parse Apache combined log format. Return object with: "
    "ip, user (null if dash), timestamp_iso (ISO 8601 with TZ), "
    "method, path, status (int), bytes (int), referrer (null if dash), user_agent."
)

def parse_line(client, line):
    r = client.post(
        "https://streamfix.dev/v1/map",
        headers={"Authorization": "Bearer sk_..."},
        json={"payload": line, "target": TARGET},
        timeout=30,
    )
    return r.json()["output"]

with httpx.Client() as client, open("access.log") as f:
    events = [parse_line(client, ln.strip()) for ln in f]

# First line: parser compiled
# Every subsequent line of same shape: cached, fast

If lines from different sources have different shapes (e.g. a mix of Apache and NGINX in the same file), they get separate cached parsers. Each unique shape costs one compile.

5. Errors and status codes

6. Limits and behavior

• Per-line vs batch: the parser is compiled once per (input shape, target text). Pass single lines for per-event processing, or pass a multi-line string for batch summaries (example 3.5).
• Sandbox: AST-validated Python. re, datetime, and collections are available; no network, filesystem, or os.
• Throughput: the endpoint is single-line per request. For very high throughput (thousands per second), use a streaming agent; see the alternatives section.

7. Alternatives and how this differs

Log parsing is a mature space. Here's how /v1/map differs structurally. For high-throughput observability pipelines, the tools below will fit better; for ad-hoc or in-app parsing, the API is simpler.

Structural difference: Vector / Logstash / Fluent Bit are pipelines (input source -> parse transform -> output sink). /v1/map is just the parse transform, called from your code. If you need the full pipeline, use one of those. If you only need the parse step inside something else you're already building, the API is simpler.

8. When NOT to use this

• You're shipping >1,000 lines/second. Use Vector or Fluent Bit. The latency overhead of an HTTP POST per line doesn't make sense at high throughput; use a streaming agent that parses in-process.
• Your log format is fully standard combined/common. Many language libraries already have a parser. They're free and run in-process.
• You already run Logstash or Elastic ingest pipelines. grok patterns are battle-tested and the path of least resistance is to add another grok step.
• You need parsing to never depend on a third-party service. The first call with each new log shape requires our service to compile the parser. Once compiled, subsequent calls run cached.