--- title: "Linux Enterprise Node Troubleshooting Guide" slug: "linux-enterprise-node-troubleshooting-guide" updated: 2025-03-07T17:40:20Z published: 2025-03-07T17:40:20Z --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.catchpoint.com/llms.txt > Use this file to discover all available pages before exploring further. # Linux Enterprise Node Troubleshooting Guide ## Catchpoint Enterprise Node utility Catchpoint Enterprise Node Instances come equipped with a utility that allows you to perform basic troubleshooting. This utility can be invoked by simply running `catchpoint` on the command line. Basic Node Instance maintenance: 1. Status of the Catchpoint Node services: `$ catchpoint status` 2. Restart Catchpoint Node services: `$ catchpoint restart` 3. Stop Catchpoint Node services: `$ catchpoint stop` 4. Start Catchpoint Node services: `$ catchpoint start` ## Log Files Log files are extremely helpful when troubleshooting a Node. Everything that the Node does in logged into a log file. These log files are located at `/var/3genlabs/hawk/syntheticnode/service/log` location. To see a list of all log files and their date and time stamp `$ ls -la /var/3genlabs/hawk/syntheticnode/service/log` Use the `less` command to open a log file (or any file) and display the information one page at a time. `$ less /var/3genlabs/hawk/syntheticnode/service/log/logfilename.log` ### **Less command - tips & tricks** By default, the less command will scroll a single page at a time. You can change the number of lines that are scrolled when you press the space and `f` key by pressing the number immediately before pressing the key. To make this number the default you can enter the number followed by the `z` key. If you want to go back to the beginning of the output press lowercase `g` and to go to the end press uppercase `G`. To go to a specific line, enter a number before pressing the `g` or `G` keys. You can search for text within the output using the `/` key followed by the text you wish to search or a regular expression. If you have finished looking at a file you can load a new file into the less command by pressing the colon key `:` followed by the `e` or `E` key and the path to a file. To exit the `less` command, press either the `q` or `Q` key. ### **Useful Command Line Switches** The following run time switches may be useful to you: 1. **less -bN** - The N stands for a number of kilobytes to load into memory. By default, the value is 64 kilobytes, but you can specify any number you wish. If you enter -1 then the entire file will be loaded into memory which may or may not be a good idea depending on the size of the file. 2. **less -B** - By default the less command allocates the required memory buffers by default when using piped output. You can use the -B switch to prevent auto buffering. 3. **less -c** or **less -C** - By default the screen repaints by scrolling up the screen. To clear the screen from the top down use the -c or -C switches. 4. **less -e** - Causes less to exit when it hits the end of the file for the second time 5. **less -E** - Causes less to exit when it hits the end of the file for the first time 6. **less -f** - Open special files such as directories using less 7. **less -F** - Causes less to exit if a file is less than one screens worth of data 8. **less -g** - Only highlight the last item found when searching 9. **less -G** - Suppress highlighting altogether when searching 10. **less -hN** - Specify the maximum number of lines the less command can scroll back 11. **less -i** - Ignore case when searching unless uppercase characters are found in the search pattern 12. **less -I** - Ignore case when searching 13. **less -jN** - The N stands for a number. This determines where on the screen a line is placed when searched for. For example, searching for "hello world" will place the line found with "hello world" in it on line 1 if less -j1 is used. 14. **less -J** - This displays a little asterisk in the left column (status column) which shows when a piece of text which you have searched for has been found. 15. **less -m** - Displays the number of bytes through a file instead of a colon at the bottom of the screen 16. **less -M** - Displays the line numbers of the output. For example, "lines 1-23" 17. **less -n** - Suppress line numbers 18. **less -N** - Display line numbers on each line 19. **less -o** - This is used with piped output only. It outputs each page of the piped output to the file one page at a time. If the file exists it will ask whether you want to overwrite it. 20. **less -O** - This is the same as -o except that it won't ask for confirmation before overwriting a file. 21. **less -p** - This starts less at the first occurrence of the pattern specified. 22. **less -P** “text” - This replaces the message at the bottom of the screen to the text specified 23. **less -q** - This prevents the bell from buzzing when you reach the end of the file. Other reasons for the bell to ring such an invalid key press remain. 24. **less -Q** - Suppresses all noises 25. **less -s** - This condenses blank lines. For example, if a file has 4 consecutive blank lines and you use the less -s command only 1 blank line will be displayed. 26. **less -S** - This causes long lines to be truncated rather than wrapping them onto the next line ## Monitoring Log Files Log files are generated at regular intervals so that the length of these log file is manageable. You can view the output of the service logs to STDOUT (similar to tail -f) by running: `$ catchpoint log` **Gathering Log Files** At times you may need to get these log files for our Engineering team to look at, and there may be many of them. To collect these log files, create a tar file first: ``` $ catchpoint log --snapshot Copying cpsns log... Copying cprapi log... Copying service management log...Copying nginx log... Copying system log... Compressing... Done. /var/tmp/catchpoint/{{hostname_timestamp}}.txz ``` The `--snapshot` or `-s` argument creates a snapshot of all logs related to Catchpoint Services and saves it to `/var/tmp/catchpoint/log/`. The output displays the complete path and the file name. By default, the above command saves the log files from the last 24 hours. If you want to gather log files for more than 24 hours: ``` $ catchpoint log --snapshot -d 2 Copying cpsns log... Copying cprapi log... Copying service management log... Copying nginx log... Copying system log... Compressing... Done. /var/tmp/catchpoint/{{hostname_timestamp}}.txz ``` The `--days` or `-d` creates a snapshot of all the logs related to Catchpoint Services for the # of days specified and saves it to `/var/tmp/catchpoint/log/`. This argument is only valid when the option `--snapshot` or `-s` is used. The default value is 1. ## Getting a TCPDump Sometimes it useful to do a TCP dump to analyze what’s going on between the Enterprise Node and the internet and/or Catchpoint C&C. To perform a TCP dump: ``` $\> catchpoint packet-capture Hit Ctrl + C to stop. tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 32 packets captured 32 packets received by filter 0 packets dropped by kernel Compressing... Done. /var/tmp/catchpoint/{{hostname_timestamp}}.cap.txz ``` The above command starts a packet capture and, when stopped, saves it as `{{Hostname_Timestamp}}.cap` in the `/var/tmp/catchpoint` folder. This `.cap` file can later be opened in a software like Wireshark for further analysis. ## Command & Control Configurations Every instance contains configuration settings that give us information as to what Command & Control servers it is communicating with. These details can be viewed by running: ``` $ catchpoint cc-config ``` ## Instance's Environment Variables Once the service is installed and communicating with our Command & Control, it will be referring to some environment variables which reflect the settings the machine/appliance is set up with. For example, Proxy settings. To view these details: ``` $ catchpoint env ``` ## Updating Enterprise Node to the Latest Version To update the Catchpoint Services to the latest version: ``` $ catchpoint update ``` ## Proxy Configuration Users can now set up a proxy on a node Instance using the command line utility. Catchpoint supports Direct, Static & PAC proxies, and each of them can be set up using a single step or multiple steps. To specify the type of proxy, use the `--type` or `-t` argument. A proxy can be either `DIRECT`, `STATIC` or `PAC`: ``` $ catchpoint proxy --type=STATIC OR $ catchpoint proxy -t STATIC ``` To set the proxy location, format depends on proxy type: `DIRECT` proxy type does not require location option. `STATIC` proxy type; supply `--host` or `--port` (Optional) for Proxy Location. For `--host` use hostname or IPv4 IP address or IPv6 IP address. ``` $ catchpoint proxy --host=your.proxy.url --port=1234 OR $ catchpoint proxy -h your.proxy.url -P 1234 ``` `PAC` - Supply the `URL` where `PAC` file using `--host` ``` $ catchpoint proxy --host=http://pac_file_url/file.p OR $ catchpoint proxy -h http://pac_file_url/file.p ``` You may have an authenticated proxy setup. You can provide the credentials as well and these are all encrypted. Proxy server Username when using a Static Proxy. ``` $ catchpoint proxy --usr={{username}} OR $ catchpoint proxy -u {{username}} ``` Proxy server password when using a Static Proxy. The password is encrypted. The password must follow `-p` without space. If no password is provided; you will be prompted for the password. ``` $ catchpoint proxy --password=\[password\] OR $ catchpoint proxy -p\[password\] ``` To reset the proxy, use the `DIRECT` option. ``` $ catchpoint proxy --type=DIRECT OR $ catchpoint proxy -t DIRECT ``` Some more examples using all parameters together: ``` $ catchpoint proxy --type=PAC --host=http://pac_file_url/file.pac \> catchpoint proxy -t PAC -h http://pac_file_url/file.pac $ catchpoint proxy --type=STATIC --host=your.proxy.url --port=1234 –-usr={{username}} --password=\[password\] $ catchpoint proxy -t STATIC -h your.proxy.url -P 1234 -u {{username}} -p\[password\] ``` ## Disable TLS 1.0 & 1.1 There's a config file that we deploy for nginx at `/etc/nginx/sites-enabled/default`. In that file, you add a new line after the `ssl_certificate_key` line as such: `ssl_protocols TLSv1.2;`. By default all three are enabled if nothing is specified. The run `sudo catchpoint restart`. To verify that it was properly disabled, run `openssl s_client -connect localhost:443 -tls1` (or `-tls1_1`). This will ensure TLS 1.0 and 1.1 is turned off. ## Troubleshooting Feature not working If you are not able to troubleshoot your Linux Node on the portal, make sure your firewall is open and SELinux is disabled. To open firewall; follow these steps: ``` $ sudo firewall-cmd --add-service=https --permanent $ sudo firewall-cmd --reload ``` To disable `SELinux`; follow these steps: Check the status of `SELinux` ``` $ sudo getenforce Enforcing ``` Set SELinux to be Permissive ``` $ sudo setenforce 0 $ sudo getenforce Permissive ``` The above steps will set `SELinux`to Permissive only temporarily until the next time your system reboots. To make this change permanent: ``` $ vi /etc/selinux/config ``` Change the setting `SELinux=Enforcing` to `SELinux=Permissive` and reboot your machine.