antman keeps crashing - java.lang.RuntimeException: could not start [#'antman.core/http-server]

Hi there,

I am running an older version of Antman (0.5.0).  Recently antman stopped working and keeps crashing.  Anthill seems to be working fine:

service anthilld status
* status: started

However, antman appears to start up, but then crashes a few minutes later:

service antman restart
* Stopping antman ...
* start-stop-daemon: no matching processes found [ ok ]
* Starting antman ... [ ok ]
# service antman status
* status: started

(wait a minute)

# service antman status
* status: crashed

The log file doesn't seem to be capturing the output correctly, so I investigated the service script: /usr/local/bin/antmand and I can see that it runs the java program:

java -jar -Xms512M -Xmx512M antman.jar

So, to debug this, I ran it from the command-line to see if I could find out the source of the error, now I'm actually getting somewhere in terms of debugging, at least:

# java -jar -Xms512M -Xmx512M /usr/local/antsleOS/antman/target/antman.jar
tool java
Exception in thread "main" java.lang.RuntimeException: could not start [#'antman.core/http-server] due to
at mount.core$up$fn__230.invoke(core.cljc:80)
at mount.core$up.invokeStatic(core.cljc:80)
at mount.core$up.invoke(core.cljc:78)
at mount.core$bring.invokeStatic(core.cljc:247)
at mount.core$bring.invoke(core.cljc:239)
at mount.core$start.invokeStatic(core.cljc:289)
at mount.core$start.doInvoke(core.cljc:281)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at mount.core$start_with_args.invokeStatic(core.cljc:389)
at mount.core$start_with_args.doInvoke(core.cljc:385)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at antman.core$start_app.invokeStatic(core.clj:77)
at antman.core$start_app.invoke(core.clj:72)
at antman.core$_main.invokeStatic(core.clj:86)
at antman.core$_main.doInvoke(core.clj:85)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at antman.core.main(Unknown Source)
Caused by: java.lang.RuntimeException: EOF while reading
at clojure.lang.Util.runtimeException(Util.java:221)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1204)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1200)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1200)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.read(LispReader.java:196)
at clojure.lang.LispReader.read(LispReader.java:185)
at clojure.lang.RT.readString(RT.java:1835)
at clojure.lang.RT.readString(RT.java:1830)
at clojure.core$read_string.invokeStatic(core.clj:3687)
at clojure.core$read_string.invoke(core.clj:3677)
at antman.util.db$db_get.invokeStatic(db.clj:15)
at antman.util.db$db_get.invoke(db.clj:13)
at antman.config$fn__460$fn__461.invoke(config.clj:18)
at clojure.lang.Delay.deref(Delay.java:37)
at clojure.core$deref.invokeStatic(core.clj:2228)
at clojure.core$deref.invoke(core.clj:2214)
at antman.middleware$wrap_base.invokeStatic(middleware.clj:102)
at antman.middleware$wrap_base.invoke(middleware.clj:101)
at antman.handler$app.invokeStatic(handler.clj:71)
at antman.handler$app.invoke(handler.clj:70)
at antman.core$fn__31039.invokeStatic(core.clj:27)
at antman.core$fn__31039.invoke(core.clj:22)
at mount.core$record_BANG_.invokeStatic(core.cljc:74)
at mount.core$record_BANG_.invoke(core.cljc:73)
at mount.core$up$fn__230.invoke(core.cljc:81)
... 18 more

But I'm still stumped.  It looks like something to do with the webserver (http), but nginx seems to be OK:

# service nginx status
* status: started

Any suggestions gratefully appreciated.

Hi @alexl

Thanks for contacting antsle Support.

Can you let us know what version of antMan you are using by typing this command below:
antman-version

Have you tried updating to the latest antMan?  You can use the instructions below:

You will need to download the antMan bundle from your anthill.antsle.com account in the Downloads page - antMan 3.4.3 for x86_64
Copy the file to your antsle using scp or an sftp client like Filezilla (https://docs.antsle.com/system/secure-ftp-sftp-access)

Or download it directly to your antsle with 'wget'. Right click the Download link and copy the link address. Then SSH to the antsles command line and run the 'wget' command and paste the link e.g:
wget https://s3.wasabisys.com/wasabi--cloud-6a2aa848-8fea-4d97-b476-2e5ef9db7c18/antman-updates/antman-3.4.3-x86_64.tar.gz

Your link may be different than this example so be sure and copy the link from antHill.

On the antsle's command line, extract the file
tar -xzf antman-X.X.X-x86_64.tar.gz

CD into the new directory
cd antman-x.x.x-x86_64

and run the installer
./install

Thank you,
antsle Support

Quote from daniel.luck on September 19, 2022, 5:50 pm

Hi @alexl

Thanks for contacting antsle Support.

Can you let us know what version of antMan you are using by typing this command below:
antman-version

Hi @daniel-luck,

Thanks for getting back to me, I tried:

antman-version

but that just gives me:

zsh: command not found: antman-version

I found antMan-version in /usr/local/bin/, but running that just shows no output.  I think I'm on version 0.5.0 (as in the thread subject), upgraded in September 2020. (Or perhaps 0.5.0 only refers to the edgeLinux version, not sure).

Have you tried updating to the latest antMan?  You can use the instructions below:

You will need to download the antMan bundle from your anthill.antsle.com account in the Downloads page - antMan 3.4.3 for x86_64
Copy the file to your antsle using scp or an sftp client like Filezilla (https://docs.antsle.com/system/secure-ftp-sftp-access)

Or download it directly to your antsle with 'wget'. Right click the Download link and copy the link address. Then SSH to the antsles command line and run the 'wget' command and paste the link e.g:
wget https://s3.wasabisys.com/wasabi--cloud-6a2aa848-8fea-4d97-b476-2e5ef9db7c18/antman-updates/antman-3.4.3-x86_64.tar.gz

Your link may be different than this example so be sure and copy the link from antHill.

On the antsle's command line, extract the file
tar -xzf antman-X.X.X-x86_64.tar.gz

CD into the new directory
cd antman-x.x.x-x86_64

and run the installer
./install

Thanks, doing an upgrade to the latest antman was going to be my next step, but I didn't want to update antman until I had done a backup - and I was trying to use antman to help with that.  So I'm in a chicken or egg situation and was hoping to fix the existing installation before doing an upgrade.  Is there nothing obvious that might be causing that java crash?

If upgrade is the only path forward, I'll do a backup outside of antman, and then try the new version using your instructions, thanks.

Hi @alexl

The only thing I found in our support articles regarding java was fix for anthilld (which is slightly different than your issue.

Shutdown inlein in case there's an inlein server running
inlein --shutdown-daemon
Then
rm /root/.inlein/port
Restart the anthilld service
service anthilld restart
If that doesn't work, upgrade to the latest anthilld version (recommended regardless if the issue was fixed or not): upgrade-anthilld

Although it's not documented it is possible to upgrade or re-install a specific version of antMan with this command:

upgrade-antman version-number
where version-number is the version you'd like to upgrade or re-install

By default, upgrade-antman will upgrade to our latest version 3.4.3
Do you remember what version of antMan you were using before?

Thanks,
antsle Support

Hi @daniel-luck

I finally got around to upgrading to the latest antMan 3.4.4 using the manual download instructions above.  I rebooted my machine, but unfortunately I still have the same error.  The java still crashes and I can't launch the web portal at <ip-address>:3000.

I then tried the inlein fix you recommended:

Shutdown inlein in case there's an inlein server running
inlein --shutdown-daemon
Then
rm /root/.inlein/port
Restart the anthilld service
service anthilld restart
If that doesn't work, upgrade to the latest anthilld version (recommended regardless if the issue was fixed or not): upgrade-anthilld

And after that it looks like at first the antman works when I run service antman status, but after a while I get:

service antman status
* status: crashed

So then I tried to also run upgrade-anthilld, and I got the following:

upgrade-anthilld
* Installing/updating antHill ...
* Download 'anthill-daemon-1.3.2.tar.gz' failed, aborting.

So I definitely know I am now running antMan 3.4.4:

antman-version
3.4.4

but I'm still stuck with the antman interface not working properly. What can I try next in the debugging?

Is this possibly related to the edgelinux version?  It looks like I'm on 0.5 (see below - I think I was previously confused about the version number of antman, 0.5 was the edgeLinux version). Does antMan 3.4.4 run on that version?  Whatever previous version of antman I had seemed to work OK.

cat /etc/edgeLinux/version
0.5.0

It also seems like I cannot connect to anthill either - "Last seen" date says "a month ago".

Quote from alexl on September 26, 2023, 4:25 pm

....

So I definitely know I am now running antMan 3.4.4:

antman-version
3.4.4

but I'm still stuck with the antman interface not working properly. What can I try next in the debugging?

Hi @daniel-luck and others watching this thread...

Yes, it's been a while - but I finally got around to working on my antlse again and managed to get antman working. So in the interests of closing the loop for anyone else who runs into the same (or similar) problem, hopefully this helps.

I started by trying to debug antman by running from the command-line to see the Java traceback, which looks like this:

exec java -jar -Xms512M -Xmx512M antman.jar

tool java
[we started again nil]
Exception in thread "main" java.lang.RuntimeException: could not start [#'antman.core/http-server] due to
at mount.core$up$fn__230.invoke(core.cljc:80)
at mount.core$up.invokeStatic(core.cljc:80)
at mount.core$up.invoke(core.cljc:78)
at mount.core$bring.invokeStatic(core.cljc:247)
at mount.core$bring.invoke(core.cljc:239)
at mount.core$start.invokeStatic(core.cljc:289)
at mount.core$start.doInvoke(core.cljc:281)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at mount.core$start_with_args.invokeStatic(core.cljc:389)
at mount.core$start_with_args.doInvoke(core.cljc:385)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at antman.core$start_app.invokeStatic(core.clj:77)
at antman.core$start_app.invoke(core.clj:72)
at antman.core$_main.invokeStatic(core.clj:87)
at antman.core$_main.doInvoke(core.clj:85)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at antman.core.main(Unknown Source)
Caused by: java.lang.RuntimeException: EOF while reading
at clojure.lang.Util.runtimeException(Util.java:221)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1204)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1200)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1200)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.read(LispReader.java:196)
at clojure.lang.LispReader.read(LispReader.java:185)
at clojure.lang.RT.readString(RT.java:1835)
at clojure.lang.RT.readString(RT.java:1830)
at clojure.core$read_string.invokeStatic(core.clj:3687)
at clojure.core$read_string.invoke(core.clj:3677)
at antman.util.db$db_get.invokeStatic(db.clj:15)
at antman.util.db$db_get.invoke(db.clj:13)
at antman.config$fn__460$fn__461.invoke(config.clj:18)
at clojure.lang.Delay.deref(Delay.java:37)
at clojure.core$deref.invokeStatic(core.clj:2228)
at clojure.core$deref.invoke(core.clj:2214)
at antman.middleware$wrap_base.invokeStatic(middleware.clj:102)
at antman.middleware$wrap_base.invoke(middleware.clj:101)
at antman.handler$app.invokeStatic(handler.clj:71)
at antman.handler$app.invoke(handler.clj:70)
at antman.core$fn__31169.invokeStatic(core.clj:27)
at antman.core$fn__31169.invoke(core.clj:22)
at mount.core$record_BANG_.invokeStatic(core.cljc:74)
at mount.core$record_BANG_.invoke(core.cljc:73)
at mount.core$up$fn__230.invoke(core.cljc:81)
... 18 more

2. After some Googling (specifically the "EOF while reading" was the clue) I realized I should be looking at a potentially corrupted file/config. So after some hunting around I found the file "db.clj" in /etc/antsleOS/ (the clj was also a clue in the traceback).

3. Sure enough the file wasn't well-formed (it was missing the closing bracket). (Perhaps the antsle was  being shutdown while antman was writing the config file? not sure when or how that happened, but it happened).

4. I moved the file aside (keeping the contents for later forensics):

cd /etc/antsleOS
mv -i db.clj db.clj.bak

5. Then I restarted antman (manually) by executing commands in sequence using the contents of /usr/local/bin/antmand

cd /usr/local/antsleOS/antman/target
export PORT=3000
if [[ -f /etc/antman.conf ]]; then
. /etc/antman.conf
fi
exec java -jar -Xms512M -Xmx512M antman.jar

6. And bingo - antman web interface is now available on the node at port 3000!

Unfortunately this meant losing some configuration (specifically port mapping) from the original installation.  To fix this I shut down the service again, then manually edited db.clj to include the configuration in a well-formed way.  For example, I could reconstruct the original rules and backup information from the old db.clj.bak file, and copied it across to the new db.clj file. (I've anonymized the unique IDs and names in the snippet below):

{
:jwt-secret "9999999999999999999999999999999"
:anonymous-id "eeeee-7777-1111-b6b6b6-11111"
:show-antman-renewal-banner? true

:nps-next -1

:pf-rules {
1 {:enabled? true, :protocol "tcp", :src-port "1014", :dst-ip "14", :dst-port "80", :dst-antlet "foo", :src-ip "99.99.99.99", :id 1}
2 {:enabled? true, :protocol "tcp", :src-port "22014", :dst-ip "14", :dst-port "22", :dst-antlet "bar", :src-ip "99.99.99.99", :id 2}
}

:scheduled-backups {
"foo" {:self-managed-enabled true}
}

:local-backups {
"9999-77777-8888-1111-2222" {:dst-path nil, :ssh-host nil, :ssh-port nil, :antlet-name "foo", :id "9999-77777-8888-1111-2222", :backend "local", :created 1662264000177}

}
}

Then I manually restarted - and now all the port rules appear in the UI again. There a maybe a few things that got missed due to the original truncation of the file - but it's good enough to get started.

Then I shutdown again and restarted using the standard: "/etc/init.d/antman start" process and all seems well again.

One suggestion to the devs (although this may be now fixed in more recent antman): there should be some way to "recover" from a bad config file - perhaps by detecting a broken config file and then restarting with a fresh config (like I did), but with a note in the UI about the corruption, so it can be "fixed" post-start. At the very least there should be some clear message somewhere in the log that this is the cause of the crash.  This would save hours of hunting around for the problem. In general, improving robustness around reading/writing the config file would be greatly appreciated.