VeilNet

Service and Logs

Find the service log on any platform, run the supervisor in the foreground, and learn when the supervisor retries or gives up.

Reading the Log

The boot service runs conflux serve, the supervisor, which starts and watches the anchor. The supervisor and the anchor daemon, anchord, write to the platform's log.

To read the last fifty lines of the log, run the command for your platform:

PlatformRead the last fifty lines
Linuxjournalctl -u conflux -n 50
macOS, FreeBSDtail -n 50 /var/log/conflux.log
OpenBSDgrep conflux /var/log/daemon | tail -n 50
WindowsGet-Content -Tail 50 "C:\ProgramData\conflux\logs\conflux.log"

When conflux up or conflux start times out, it prints the log command for this machine. To look up a message from the log, see Anchor Start Failures.

Running in the Foreground

If the service starts but the anchor does not, run the supervisor in the foreground to watch it. In the foreground, lines from anchord carry the prefix anchord:, so you can read its own reason for a refusal.

0 of 3 done

  1. Stop the service, so that the foreground run is the only supervisor. The service keeps its registration and configuration:

    $ sudo conflux down

    On Linux, sudo systemctl stop conflux does the same.

Retrying and Giving Up

At every start, the supervisor reads conflux.json, renews the credential if renewal is due, and starts the anchor.

Fig. 01 — The four outcomes of a start
THE SERVICE MANAGER RUNSconflux serveTHE SUPERVISOR, EVERY STARTBring up the anchorEXTRACT · TOKEN · ANCHORD · RENEW IF DUE · STARTIT CAME UPWatch itCHECK EVERY 30 S,REBUILD IF GONETRANSIENTRetryBACK OFF FROM 1 S,DOUBLING TO 30 SPERMANENTGive upAFTER 3 ATTEMPTS:EXIT 70NO CONFIGURATIONStopNOTHING TO RUN:EXIT 78

The service manager runs conflux serve, the supervisor, which brings the anchor up. The supervisor retries a transient failure with a growing backoff. After three attempts, a failure that no retry can change stops the supervisor with exit code 70. On a machine with no configuration, the supervisor stops immediately with exit code 78. Any other stop exits with code 1, and the service manager restarts the supervisor.

The supervisor handles a failed start in one of four ways:

  • A transient failure is retried, with a backoff of up to thirty seconds. This lets a --subnet entry succeed at boot even if its interface is not up yet.
  • A failure that no retry can change stops the supervisor after three attempts, with exit code 70.
  • If there is nothing to start, the supervisor exits with code 78.
  • Any other stop, such as a failed extraction onto a full disk, exits with code 1. The service manager then restarts the supervisor.

These failures stop the supervisor with exit code 70:

  • A configuration or credential that conflux refuses, including taints that the credential does not grant
  • An expired credential with no renewal address, or with no AnchorID recorded to renew it with
  • A credential for another realm tree
  • A TUN device that the host will not provide
  • A host that cannot be set up to forward traffic
  • Arguments that anchorctl refuses
What each platform does after exit 70 or 78
PlatformRegistrationAfter exit 70 or 78
Linux/etc/systemd/system/conflux.service, Type=notify, Restart=on-failure, RestartPreventExitStatus=78 70The unit stays in the failed state, as systemctl status conflux shows.
macOS/Library/LaunchDaemons/org.veilnet.conflux.plist, KeepAlive on unsuccessful exitlaunchd cannot exempt exit codes, so it restarts the supervisor after its throttle interval.
WindowsService conflux, recovery after 5, 10 and 30 secondsThe service is reported as stopped, not failed, so recovery does not restart it.
FreeBSD/usr/local/etc/rc.d/conflux, run under daemon(8)Stays down.
OpenBSD/etc/rc.d/conflux, enabled with rcctlStays down.

Fix the problem that the log names, then run sudo conflux start.

On systemd, systemctl start conflux returns only once the anchor is up. A unit ordered After=conflux.service can therefore rely on the anchor being up.

Watching the Anchor

Anchord can lose its anchor without exiting. The supervisor checks every thirty seconds, or every ten seconds on a machine with an uplink, and rebuilds the anchor if it is gone.

  • If a rebuild fails permanently three times in a row, the supervisor gives up and records this in the log. One cause is a conflux.json edited so that the anchor can no longer be built. The service keeps running without an anchor, and renews nothing, until you run sudo conflux start or reboot the machine.
  • The supervisor does not rebuild a blocked anchor. A blocked anchor keeps running outside its realm, so a rebuild would change nothing.

Collecting a Bug Report

If the log does not explain the fault, collect the output of these commands and send it in one report:

$ conflux version
$ sudo conflux status
$ sudo journalctl -u conflux -n 100 --no-pager
$ sudo conflux peers
$ sudo conflux routes

conflux version prints the SHA-256 digests of both anchor binaries, which identify the exact build. The journalctl command is for Linux. On other platforms, take the last hundred lines of the log with the command from Reading the Log.

None of this output contains key material. Send it through Report a problem on the pre-release page.