[SOLVED] Bizarre Antsle Network Issues - Can't update, Can't Port Forward

Running the Antsle (edgeLinux 0.5.0) on my home network. I know it cannot be related, but as soon as I patched and reboot a Ubuntu LXC I started seeing weird network issues with my Antsle:

• I lost http connectivity from my wifi enabled laptop to AntMan (0.11.97947208069579961a) that I was working from.
• I cannnot connect to my Antsle via a ssh connection directly from my Wifi laptop.
• I cannot connect to any Antlets via ssh (-p 22xxx)
• I can ssh to Antlets over 192.168.1.x on my local NAT'd network.
• From an Antlet, I can connect to the Antsle over 10.1.1.x
• If I use a computer with a wired network connection, I can connect to AntMan and ssh directly with no issues.
• While connected via ssh to the Antsle with a shell, I cannot ping out to the Internet, only devices within the local network.
• All my Internet based websites that my router port forwards through the Antsle do not connect, but they are available from the NAT'd 192.168.x.x network.
• There have been no changes to my router in over a month
• All Antlets seem to be performing as they all should. They are having no network issues.

From the Antsle, if I login with a shell and ping something on the internet, it resolves to my router. I wish nslookup or dig was installed. I can connect to http with the nc command to my web servers on the Local Network, but not using DNS. Other systems point resolv.conf to my router and resolve DNS just fine.

# ping yahoo.com
PING yahoo.com (192.168.1.1) 56(84) bytes of data.

# cat /etc/resov.conf

nameserver 192.168.1.1

Here's the routing table.

Kernel IP routing table
Destination	Gateway	Genmask	Flags	Metric	Ref	Use	Iface
default	localhost	0.0.0.0	UG	8	0	0	br0
localhost	0.0.0.0	255.255.255.0	U	0	0	0	virbr1
localhost	0.0.0.0	255.255.255.0	U	0	0	0	virbr2
localhost	0.0.0.0	255.255.255.0	U	8	0	0	br0
localhost	0.0.0.0	255.255.255.0	U	0	0	0	virbr0

I also thought if I try to upgrade antman, it may clean something up, and I get the following error. I suspect cause it can't get out to the internet. An upgrade of edgeLinux yields similar results.

# upgrade-antman Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python2.7/json/__init__.py", line 291, in load **kw) File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded --2019-10-05 14:29:08-- http://central.antsle.com/antman-releases//amanupd- Resolving central.antsle.com... 192.168.1.1 Connecting to central.antsle.com|192.168.1.1|:80... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2019-10-05 14:29:08 ERROR 503: Service Unavailable.

chmod: cannot access 'amanupd-': No such file or directory /usr/local/bin/upgrade-antman: line 11: ./amanupd-: No such file or directory rm: cannot remove 'amanupd-': No such file or directory Error while upgrading antman * WARNING: antman has already been started

Any help on a direction to check is GREATLY appreciated!

Hi @akak01000101,

Looks like something is blocking central.antsle.com, please try:

curl http://central.antsle.com/api/antman-info

The response should be a JSON string that looks like this:

{"version":"2.1.0c","edgeLinux-version":"2.1.0c","anthilld-version":"1.2.7","early-access-promo-due":1535525940000}

Do you have a firewall blocking central.antsle.com?

Multiple systems can connect through the firewall no problem to central.antsle.com and return:

{"version":"2.1.0c","edgeLinux-version":"2.1.0c","anthilld-version":"1.2.7","early-access-promo-due":1535525940000}

I'm thinking a routing issue somewhere in the Antsle:

root@myantsle:/tmp # wget http://central.antsle.com/api/antman-info
--2019-10-08 12:51:32-- http://central.antsle.com/api/antman-info
Resolving central.antsle.com... 192.168.1.1
Connecting to central.antsle.com|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2019-10-08 12:51:34-- (try: 2) http://central.antsle.com/api/antman-info
Connecting to central.antsle.com|192.168.1.1|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2019-10-08 12:51:34 ERROR 503: Service Unavailable.

root@myantsle:/tmp # curl -vvv http://central.antsle.com/api/antman-info
* Trying 192.168.1.1...
* TCP_NODELAY set
* Connected to central.antsle.com (192.168.1.1) port 80 (#0)
> GET /api/antman-info HTTP/1.1
> Host: central.antsle.com
> User-Agent: curl/7.54.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host central.antsle.com left intact
curl: (52) Empty reply from server

Hi @akak01000101,

Looking at the curl command output, I think the `upgrade-antman` command will work now. If so, could you please check if the issues you listed above still happen?

Thanks Mardo! Looks like your recommendation worked when I comment out the following lines in /etc/conf.d/net

Restarting the net.br0 was a temporary fix. To truly fix the issue I needed to do a complete reset of edgeLinux. Instructions below were provided by Antsle support:

CLICK ON THIS WINDOW TO EXPAND THE INSTRUCTIONS

The following instructions how to reset the edgeLinux OS to version 0.12.1. antlets will remain and the root password will be set to 'antsle'.

**This will remove all docker images and containers.

Using 2 USB sticks is the easiest method. The first will be a bootable USB stick, we recommend using SystemRescueCD:
http://www.system-rescue-cd.org/Installing-SystemRescueCd-on-a-USB-stick/
I use a utility called Rufus to create the bootable usb with the sysrescue .iso (https://rufus.akeo.ie)

The second should be at least 32G capacity and formatted with NTFS or exfat file system as one of the uncompressed images is ~16G.

Connect a monitor and keyboard or connect to the terminal via IPMI: https://docs.antsle.com/setup/#4-ipmi

Remove all other USB storage devices.
Boot the antsle from the bootable USB.
Run the command

lsblk

Your bootable USB should be the last device (sdc)
Plug in the 2nd USB and run 'lsblk' again. This USB should now be the last device (sdd). This USB will probably only have 1 partition but may have two. We will need the one using all the storage space (see the 'size' column). I'll use 'sdd1' in the example.

Lets mount the 2nd USB

mkdir /usb2
mount /dev/sdd1 /usb2

Change into the /usb2 directory

cd /usb2

Download the reset script, make it executable, and run the script:

wget https://static-files.antsle.com:8443/edgeLinux/0.12.1/el-reset
chmod +x el-reset
./el-reset

When the script finishes, power off the antsle, remove the USB sticks and reboot.

** The new root password will be 'antsle'.

Login at the prompt and run the command

el-reset-phase2