BUGLYST.
Practice

Docker · advanced

Docker service works in container but not from host: trace published address

When a Docker container reports a service listening inside its namespace but the host cannot reach the published port, the failure almost always lives in the publication path between the container's network namespace and the host's listener, not in the application itself. This guide frames the bug as a binding-vs-publication discrepancy and walks engineers from observable symptom to a verified fix using only read-only Docker commands and generic reachability checks.

The symptoms

  • •`docker ps` shows the container as Up with a `-p <host_port>:<container_port>` mapping, yet connections from the host to <host_port> fail with connection refused or timeout.
  • •From inside the container, `ss -ltn` or the process's own bind log shows the service listening on 127.0.0.1 or the container IP, but only the in-namespace listener is reachable via `docker exec`.
  • •Restarting the container, rebuilding the image, or restarting the daemon does not change the symptom, which rules out a transient process or daemon fault.
  • •A second container on the same network reaches the service on the container's internal IP and port, while the host sees nothing, indicating the publication boundary is broken rather than the application layer.
  • •`docker inspect` reports a populated `HostConfig.PortBindings` or `NetworkSettings.Ports` entry, but a host-side socket listener on the host port is absent in `ss -ltn` output for the Docker-managed interfaces.

Likely causes

  • •The application binds to `127.0.0.1` (loopback) inside the container, so only in-namespace connections succeed; host port publication is correct but there is nothing to forward to.
  • •The Dockerfile or compose file uses `network_mode: host` (or `--network host`) while the operator assumes the bridge publication path is active, so no `-p` translation exists.
  • •The published port collides with an existing host listener or a stale Docker proxy from a previous container holding the same host port.
  • •A custom network is attached after the container is created, or multiple networks are attached, and port publication is only applied to the default bridge, leaving the requested interface unrouted.
  • •Port mapping specifies the container port as `udp` while the service listens on `tcp` (or vice versa), so the kernel publishes a different transport than the application exposes.
  • •Userland proxy is disabled in daemon configuration and the host's kernel port-forwarding table lacks the corresponding entry, so the bridge has no listener on the host port.

First ten minutes

  1. 01Confirm scope: capture the container ID, image, and the exact `-p` or compose mapping from `docker ps --no-trunc`; record the failure timestamp from the host-side reachability check.
  2. 02Compare in-namespace and host-side listeners side by side: `docker exec <container> ss -ltnp` versus `ss -ltnp` on the host, looking for the host port under docker-proxy or the bridge's address.
  3. 03Read the application's bind line from inside the container only: `docker exec <container> cat /proc/<pid>/net/tcp` and the process's own log to see whether it bound to 0.0.0.0, the container IP, or 127.0.0.1.
  4. 04Inspect the container's resolved publication state with `docker inspect --format '{{json .NetworkSettings.Ports}}' <container>` and compare it to `HostConfig.PortBindings`; mismatch indicates a publication-time discrepancy.
  5. 05Run a generic reachability check from the host's bridge subnet to the container IP and container port (not loopback), then from another container on the same network; the contrast isolates the publication boundary.
  6. 06Decide before changing anything: if the in-namespace listener is on loopback, the bug is the bind address; if the host listener is absent, the bug is publication; if both look right, escalate to daemon proxy and iptables evidence.

Evidence to collect

  • •`docker ps --no-trunc` output showing the container status, image, and the literal `-p` flag as parsed by the daemon.
  • •`docker inspect` fields `HostConfig.PortBindings`, `HostConfig.NetworkMode`, and `NetworkSettings.Ports` for the container, captured as JSON.
  • •Inside the container: the process's listening sockets from `ss -ltnp` (or `/proc/<pid>/net/tcp`) and any bind or startup log lines that name the bound address.
  • •On the host: `ss -ltnp` output filtered for the host port, plus the presence of `docker-proxy` processes owning that socket.
  • •On the host: `iptables -t nat -S` (read-only) to verify DNAT rules for the host port exist for the container's IP, and `iptables -t nat -L -n -v` to confirm rule hit counts when a test connection is attempted.
  • •Generic reachability evidence: a failed host-side connection attempt with the kernel error code (connection refused vs no route), and a successful peer-container connection to the container IP plus container port.

Where to look

  • •The Docker bridge boundary, specifically `docker0` (or the user-defined bridge) and the `docker-proxy` listeners attached to host interfaces.
  • •The container's network namespace, where the application's actual bind address lives and can be observed via `docker exec`.
  • •The host's NAT table and forwarding rules, where published ports are translated from host port to container IP plus container port.
  • •The daemon's effective configuration, where settings such as `userland-proxy`, `ip-forward`, and `iptables` decide whether the publication path is wired at all.
  • •The compose file or run command source that produced the container, since publication is fixed at creation time and later edits to a running container do not retroactively apply.

Diagnostic steps

  1. 01Reproduce the symptom by attempting a generic TCP connect from the host to the published host port while watching `ss -ltnp` on the host; a connection refused with no SYN reaching `docker-proxy` points to a missing host listener, while a timeout with a SYN observed points to a NAT/forwarding gap.
  2. 02From `docker exec`, run `ss -ltnp` and record the local address bound by the service; if it is 127.0.0.1 or ::1, the bug is the bind address regardless of how publication is configured.
  3. 03Compare `docker inspect` `HostConfig.PortBindings` to `NetworkSettings.Ports`: identical keys and values mean the daemon acknowledged the mapping; missing keys mean the request was rejected or overwritten.
  4. 04Check `iptables -t nat -S` for a DNAT rule from the host port to the container IP plus container port; absence rules out kernel forwarding, presence with zero hits on a test connect rules out the application listener.
  5. 05From a second container on the same user-defined network, attempt a connect to the container's IP plus container port; success confirms the application is healthy and isolates the failure to the host publication path.
  6. 06If `NetworkMode` is `host`, all port-mapping fields are inert; the symptom then reduces to whether the process bound to a host-reachable address inside the container's namespace, which equals the host's namespace in that mode.
  7. 07Cross-check the transport in the publish spec: a `udp` mapping against a `tcp` listener (or the reverse) produces a published socket that does not match the application's protocol and will appear as unreachable from a mismatched client.

Common mistakes

  • •Assuming `EXPOSE` in the Dockerfile publishes a port; `EXPOSE` is documentation metadata and creates no host listener unless paired with `-p` at run time.
  • •Assuming restarting the container rewrites publication; publication is applied at container create, so changes to `-p` after creation are not honored until a new container is created.
  • •Adding `-p` to a container started with `--network host`; host networking bypasses publication entirely, so the flag has no effect and the symptom persists.
  • •Trusting a passing `docker inspect` Ports field as proof of reachability; the field is a request echoed back, not a guarantee that a host listener exists or that the application bound correctly.
  • •Replacing the image or rebuilding before confirming the bind address inside the container; if the application is binding to loopback, no image change will fix the publication boundary.
  • •Disabling the userland proxy as a first move when the real defect is a loopback bind; the symptom will reappear immediately on the next container and the daemon change adds noise to the evidence.

Safe fixes

  • •If `docker exec <container> ss -ltnp` shows the service bound to 127.0.0.1, change the application's bind configuration to 0.0.0.0 (or the container's interface address) and restart only the process inside the container; then re-run the host-side reachability check before any container recreation.
  • •If `HostConfig.PortBindings` is empty while `NetworkSettings.Ports` is populated, or vice versa, stop and recreate the container with an explicit `-p <host_port>:<container_port>` flag carried over from the original mapping; do not edit a running container in place.
  • •If `NetworkMode` is `host`, remove the `-p` expectation, document the host-mode behavior, and instead bind the application to the host-reachable address it is intended to serve on; verify with `ss -ltnp` on the host directly.
  • •If the transport in the publish spec is wrong (tcp vs udp), recreate the container with the corrected mapping; do not attempt to patch the transport on a running container.
  • •If the host port collides with an existing listener or a stale `docker-proxy`, choose a different host port in the recreation and verify that no other process owns it via `ss -ltnp` before relaunching.
  • •If the userland proxy is disabled at the daemon level and kernel forwarding shows no DNAT rule, treat this as a daemon-configuration decision, not an application bug; resolve via documented daemon settings and restart the affected container so publication is reapplied.

Prove the fix

  1. 01`docker exec <container> ss -ltnp` shows the service bound to a non-loopback address inside the namespace, and the same port is visible in `ss -ltnp` on the host under a `docker-proxy` owner or the bridge address.
  2. 02`docker inspect --format '{{json .NetworkSettings.Ports}}' <container>` returns the expected mapping for the published host port and container port, and a generic host-side connect attempt to the host port completes the TCP handshake.
  3. 03`iptables -t nat -L -n -v` shows non-zero packet and byte counters on the DNAT rule for the published port after a single test connection, confirming the publication path is exercised end to end.
  4. 04A peer container on the same network connects to the container IP plus container port, and the host connects to the host port, with both reaching the same application instance, demonstrating symmetric reachability across the publication boundary.
  5. 05The fix is recorded with the original symptom capture and the post-fix listener evidence so the next operator can distinguish a regression in the bind address from a regression in publication.

Prevention and next steps

  • •Make the application's bind address an explicit configuration value, default it to 0.0.0.0 inside containers, and reject 127.0.0.1 in container images that are intended to be reachable from the host.
  • •Treat `-p` mappings as create-time contracts: store the exact run command or compose excerpt alongside the container ID so future recreations do not drift on host port, container port, or transport.
  • •Add a post-create smoke check that runs a generic host-side connect against the published host port and records the result alongside `docker inspect` Ports output, so publication regressions are caught before traffic is routed.
  • •Avoid mixing `--network host` with `-p` in runbooks and compose files; pick one model per service and document the choice near the service definition.
  • •When changing daemon-level settings such as the userland proxy or iptables integration, restart affected containers in a controlled window so publication is reapplied against the new daemon state and the change is observable in `iptables -t nat -S`.

Safe commands and checks

docker ps --no-trunc
docker inspect --format '{{json .NetworkSettings.Ports}}' <container>
docker inspect --format '{{json .HostConfig.PortBindings}}' <container>
docker inspect --format '{{json .HostConfig.NetworkMode}}' <container>
docker exec <container> ss -ltnp
ss -ltnp
iptables -t nat -S
iptables -t nat -L -n -v