Troubleshooting the client
Symptoms, the likely cause, and the command that tells them apart.
Start with sxaccess info -d every time. It answers the first questions - is the agent running, does this device reach the SXAccess services, and what does it know about each device it may connect with - and most faults are decided by that output alone. If the agent is not running, the command says so instead of printing anything else.
Reading the detail view
The first line is the answer. The rows under it are the parts that produced it, followed by a Devices: list with one block for each device an access rule connects this one with, and the agent's recent events.
| Line | What it tells you |
|---|---|
Connected, Partly connected, Not connected | Whether this device reaches both the management service and Discovery. Partly connected means one of them answers and the other does not, and the row of the one that does not says why |
This device, Address | The device's DNS name and its overlay address |
Devices, Networks | How many of the devices this one may connect with are connected now, for example 3 of 5 connected, and the routed networks it uses |
Service, Discovery | Connected to and the endpoint, or Disconnected with the reason in brackets. Service is the management service at mgmt.sxaccess.com |
Relays | Every STUN, TURN and relay endpoint the management service handed to this device, each marked Reachable or Not reachable |
DNS servers | Each DNS server that applies to this device, the domains it answers for, and whether it is reachable |
Interface | Kernel or Userspace: where the tunnel runs. Only the agent on Linux can use the kernel |
Identity, Quantum res., Lazy conn. | The identity in use, and whether quantum resistance and on-demand connections are on |
Agent, CLI | The version of the running agent and of the sxaccess command. Quote both when you ask for support |
Path, per device | P2P for a direct path, Relayed for one through a relay |
Candidates, Round trip, Last handshake, Transferred, Reaches, per device | The kind of address each end uses, the latency, when the tunnel last completed a handshake, the bytes received and sent, and the routed networks reached through that device |
Symptom to cause
| Symptom | Likely cause | What to do |
|---|---|---|
info prints cannot reach the SXAccess agent | The agent service is stopped, or was never installed | sxaccess agent info prints Agent service: Running or Stopped. Start it with sudo sxaccess agent start, then connect with sxaccess start. If it was never installed, run sudo sxaccess agent install first |
info prints This device is not signed in, or the device worked for a day and then stopped | Its session expired. Only devices enrolled by signing in have a session, 24 hours by default; devices enrolled with an enrollment key never expire. A deleted device, or one that belongs to a blocked member, gets the same message | sxaccess start signs it in again; enrol a server again with an enrollment key. The length is set under Settings → Security → Session length, see Organization settings |
info prints This device's organization is suspended … | Access is paused for the whole organization, not for this device | Nothing to do on the device: the agent tries again about every 5 minutes and reconnects on its own once the organization is active again, without a new sign-in. Contact your organization's administrator, and see Organizations |
info says Connected, but a device you expect is not in the list | No enabled access rule connects the two devices, in either direction, or a device requirement on the rule leaves the source device out. A device that fails a rule's device requirements is dropped from that rule without a message | Check both devices' groups and the access rules, then the rule's device requirements |
| A device is listed but unreachable | A rule connects the devices, but not for the protocol or port you are using, or nothing listens on that port | Trace the connection on the device you are trying to reach. Trace why a connection is blocked shows which stage drops it |
Every device shows Path: Relayed | Outbound UDP is blocked, or both ends sit behind strict NAT | See NAT and connectivity and Restricted networks. The Relays block of sxaccess info -d shows which STUN and TURN endpoints this device reaches |
| Device disappeared from the dashboard | It was enrolled with a key that has Temporary devices on, and it was offline for more than 10 minutes | Enrol it again. For a long-lived machine, use a key without Temporary devices; see Enrollment keys |
| Internal names do not resolve | No DNS server applies to this device for that domain, DNS management is turned off for it, or another resolver on the machine answers first | Check the DNS servers block of sxaccess info -d, and see DNS. On Linux, check what manages /etc/resolv.conf |
| A routed network is not reachable | The network is published to this device but not selected on it | sxaccess reach list shows each network with state: Selected or Not Selected. sxaccess reach add -a <name> adds one, by the name reach list shows; without -a, add replaces the current selection. A network missing from the list is not published to this device: check the access rule that names the resource, or the route's Applies to groups |
| After a crash or power loss, DNS or routes still point at SXAccess while it is disconnected | The agent did not get to undo the changes it made | sxaccess stop, then sxaccess store reset --all. See Stored state |
| Tunnel is up, no traffic passes | Another VPN or a security product's network filter handles the traffic first | Turn the other product off and test again |
Reading the logs
The agent logs to /var/log/sxaccess/client.log on Linux and macOS. The file belongs to root, so read it with sudo. At info, the default, it records enrolment, map updates and connection changes. At debug it adds ICE candidate gathering and every connection attempt, which is where the answer usually is; trace is more detailed still. sxaccess diagnose verbosity changes the level until the agent restarts.
sxaccess diagnose verbosity debugsudo tail -f /var/log/sxaccess/client.logPress Ctrl-C to stop following the log. To go back to the normal level, run the command below; restarting the agent does the same.
sxaccess diagnose verbosity infoCollecting a bundle
For anything you cannot resolve yourself, send a diagnostics bundle. sxaccess diagnose collect packs the agent's logs, the detail view, its settings, the network map it last received and the system's routes, interfaces and, on Linux, firewall rules into one archive. -S=false leaves the system information out, and -C sets how many older log files go in (one by default).
sxaccess diagnose collectFor a fault that only happens sometimes, sxaccess diagnose capture 5m is the right tool. It switches the agent to trace logging, disconnects and reconnects the device so the log starts from a fresh connection, waits the time you give, builds the archive and puts the level back. Because it reconnects, sessions running over SXAccess, such as SSH, drop. A device that was disconnected is connected for the capture and disconnected again afterwards.
sxaccess diagnose capture 5mBoth print Bundle saved to: and the path of a sxaccess-support-*.zip file in the agent's temporary directory. Add -A to either command to replace public addresses and outside domains first, as in sxaccess diagnose collect -A. Only root can read the archive, so copy it somewhere you can read it before you attach it, with its path in place of <path>:
sudo cp <path> ~/ && sudo chown "$USER" ~/sxaccess-support-*.zipStarting clean
When a machine's local state is suspect, start it again from nothing. The commands below work on Linux and macOS, and they are the same steps as Resetting the agent. The machine comes back as a new device and may get a different overlay address, so update anything that referred to the old one.
First disconnect, undo the DNS and route changes the agent recorded, and stop the agent:
sxaccess stopsxaccess store reset --allsudo sxaccess agent stopNow delete the device under Devices in the dashboard. That revokes its key, and it frees the machine's DNS name: while the old entry exists, a new device gets that name with two numbers added.
Then delete the agent's state, which the agent creates again when it starts, and your own folder that records which identity the CLI uses: ~/.config/sxaccess on Linux, ~/Library/Application Support/sxaccess on macOS. Start the agent and enrol again. On a server, end with sxaccess start --enrollment-key-file <file> instead.
rm -rf ~/.config/sxaccess "$HOME/Library/Application Support/sxaccess"sudo rm -rf /var/lib/sxaccess && sudo sxaccess agent start && sxaccess start