Antsle Forum

Welcome to our Antsle community! This forum is to connect all Antsle users to post experiences, make user-generated content available for the entire community and more. 

Please note: This forum is about discussing one specific issue at a time. No generalizations. No judgments. Please check the Forum Rules before posting. If you have specific questions about your Antsle and expect a response from our team directly, please continue to use the appropriate channels (email: [email protected]) so every inquiry is tracked. 

Please or Register to create posts and topics.

antman keeps crashing - java.lang.RuntimeException: could not start [#'antman.core/http-server]

alexl@alexl
4 Posts
#1 · September 19, 2022, 3:37 pm

Hi there,

I am running an older version of Antman (0.5.0).  Recently antman stopped working and keeps crashing.  Anthill seems to be working fine:

service anthilld status
* status: started

However, antman appears to start up, but then crashes a few minutes later:

service antman restart
* Stopping antman ...
* start-stop-daemon: no matching processes found [ ok ]
* Starting antman ... [ ok ]
# service antman status
* status: started

(wait a minute)

# service antman status
* status: crashed

The log file doesn't seem to be capturing the output correctly, so I investigated the service script: /usr/local/bin/antmand and I can see that it runs the java program:

java -jar -Xms512M -Xmx512M antman.jar

So, to debug this, I ran it from the command-line to see if I could find out the source of the error, now I'm actually getting somewhere in terms of debugging, at least:

# java -jar -Xms512M -Xmx512M /usr/local/antsleOS/antman/target/antman.jar
tool java
Exception in thread "main" java.lang.RuntimeException: could not start [#'antman.core/http-server] due to
at mount.core$up$fn__230.invoke(core.cljc:80)
at mount.core$up.invokeStatic(core.cljc:80)
at mount.core$up.invoke(core.cljc:78)
at mount.core$bring.invokeStatic(core.cljc:247)
at mount.core$bring.invoke(core.cljc:239)
at mount.core$start.invokeStatic(core.cljc:289)
at mount.core$start.doInvoke(core.cljc:281)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at mount.core$start_with_args.invokeStatic(core.cljc:389)
at mount.core$start_with_args.doInvoke(core.cljc:385)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at antman.core$start_app.invokeStatic(core.clj:77)
at antman.core$start_app.invoke(core.clj:72)
at antman.core$_main.invokeStatic(core.clj:86)
at antman.core$_main.doInvoke(core.clj:85)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at antman.core.main(Unknown Source)
Caused by: java.lang.RuntimeException: EOF while reading
at clojure.lang.Util.runtimeException(Util.java:221)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1204)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1200)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.readDelimitedList(LispReader.java:1200)
at clojure.lang.LispReader$MapReader.invoke(LispReader.java:1158)
at clojure.lang.LispReader.read(LispReader.java:263)
at clojure.lang.LispReader.read(LispReader.java:196)
at clojure.lang.LispReader.read(LispReader.java:185)
at clojure.lang.RT.readString(RT.java:1835)
at clojure.lang.RT.readString(RT.java:1830)
at clojure.core$read_string.invokeStatic(core.clj:3687)
at clojure.core$read_string.invoke(core.clj:3677)
at antman.util.db$db_get.invokeStatic(db.clj:15)
at antman.util.db$db_get.invoke(db.clj:13)
at antman.config$fn__460$fn__461.invoke(config.clj:18)
at clojure.lang.Delay.deref(Delay.java:37)
at clojure.core$deref.invokeStatic(core.clj:2228)
at clojure.core$deref.invoke(core.clj:2214)
at antman.middleware$wrap_base.invokeStatic(middleware.clj:102)
at antman.middleware$wrap_base.invoke(middleware.clj:101)
at antman.handler$app.invokeStatic(handler.clj:71)
at antman.handler$app.invoke(handler.clj:70)
at antman.core$fn__31039.invokeStatic(core.clj:27)
at antman.core$fn__31039.invoke(core.clj:22)
at mount.core$record_BANG_.invokeStatic(core.cljc:74)
at mount.core$record_BANG_.invoke(core.cljc:73)
at mount.core$up$fn__230.invoke(core.cljc:81)
... 18 more

But I'm still stumped.  It looks like something to do with the webserver (http), but nginx seems to be OK:

# service nginx status
* status: started

Any suggestions gratefully appreciated.

Click for thumbs down.0Click for thumbs up.0
Last edited on September 26, 2023, 4:27 pm by alexl
daniel.luck@daniel-luck
552 Posts
Antsle Staff
#2 · September 19, 2022, 5:50 pm

Hi @alexl

Thanks for contacting antsle Support.

Can you let us know what version of antMan you are using by typing this command below:
antman-version

Have you tried updating to the latest antMan?  You can use the instructions below:

You will need to download the antMan bundle from your anthill.antsle.com account in the Downloads page - antMan 3.4.3 for x86_64
Copy the file to your antsle using scp or an sftp client like Filezilla (https://docs.antsle.com/system/secure-ftp-sftp-access)

Or download it directly to your antsle with 'wget'. Right click the Download link and copy the link address. Then SSH to the antsles command line and run the 'wget' command and paste the link e.g:
wget https://s3.wasabisys.com/wasabi--cloud-6a2aa848-8fea-4d97-b476-2e5ef9db7c18/antman-updates/antman-3.4.3-x86_64.tar.gz

Your link may be different than this example so be sure and copy the link from antHill.

On the antsle's command line, extract the file
tar -xzf antman-X.X.X-x86_64.tar.gz

CD into the new directory
cd antman-x.x.x-x86_64

and run the installer
./install

Thank you,
antsle Support

Click for thumbs down.0Click for thumbs up.1
Last edited on September 19, 2022, 5:50 pm by daniel.luck
alexl has reacted to this post.
alexl
alexl@alexl
4 PostsTopic Author
#3 · September 20, 2022, 2:36 pm
Quote from daniel.luck on September 19, 2022, 5:50 pm

Hi @alexl

Thanks for contacting antsle Support.

Can you let us know what version of antMan you are using by typing this command below:
antman-version

Hi @daniel-luck,

Thanks for getting back to me, I tried:

antman-version

but that just gives me:

zsh: command not found: antman-version

I found antMan-version in /usr/local/bin/, but running that just shows no output.  I think I'm on version 0.5.0 (as in the thread subject), upgraded in September 2020. (Or perhaps 0.5.0 only refers to the edgeLinux version, not sure).

Have you tried updating to the latest antMan?  You can use the instructions below:

You will need to download the antMan bundle from your anthill.antsle.com account in the Downloads page - antMan 3.4.3 for x86_64
Copy the file to your antsle using scp or an sftp client like Filezilla (https://docs.antsle.com/system/secure-ftp-sftp-access)

Or download it directly to your antsle with 'wget'. Right click the Download link and copy the link address. Then SSH to the antsles command line and run the 'wget' command and paste the link e.g:
wget https://s3.wasabisys.com/wasabi--cloud-6a2aa848-8fea-4d97-b476-2e5ef9db7c18/antman-updates/antman-3.4.3-x86_64.tar.gz

Your link may be different than this example so be sure and copy the link from antHill.

On the antsle's command line, extract the file
tar -xzf antman-X.X.X-x86_64.tar.gz

CD into the new directory
cd antman-x.x.x-x86_64

and run the installer
./install

 

Thanks, doing an upgrade to the latest antman was going to be my next step, but I didn't want to update antman until I had done a backup - and I was trying to use antman to help with that.  So I'm in a chicken or egg situation and was hoping to fix the existing installation before doing an upgrade.  Is there nothing obvious that might be causing that java crash?

If upgrade is the only path forward, I'll do a backup outside of antman, and then try the new version using your instructions, thanks.

Click for thumbs down.0Click for thumbs up.0
daniel.luck@daniel-luck
552 Posts
Antsle Staff
#4 · September 20, 2022, 5:12 pm

Hi @alexl

The only thing I found in our support articles regarding java was fix for anthilld (which is slightly different than your issue.

Shutdown inlein in case there's an inlein server running
inlein --shutdown-daemon
Then
rm /root/.inlein/port
Restart the anthilld service
service anthilld restart
If that doesn't work, upgrade to the latest anthilld version (recommended regardless if the issue was fixed or not): upgrade-anthilld

Although it's not documented it is possible to upgrade or re-install a specific version of antMan with this command:

upgrade-antman version-number
where version-number is the version you'd like to upgrade or re-install

By default, upgrade-antman will upgrade to our latest version 3.4.3
Do you remember what version of antMan you were using before?

Thanks,
antsle Support

Click for thumbs down.0Click for thumbs up.1
Last edited on September 20, 2022, 5:15 pm by daniel.luck
alexl has reacted to this post.
alexl
alexl@alexl
4 PostsTopic Author
#5 · September 26, 2023, 4:25 pm

Hi @daniel-luck

I finally got around to upgrading to the latest antman 3.4.4 using the manual download instructions above.  I rebooted my machine, but unfortunately I still have the same error.  The java still crashes and I can't launch the web portal at <ip-address>:3000.

I then tried the inlein fix you recommended:

Shutdown inlein in case there's an inlein server running
inlein --shutdown-daemon
Then
rm /root/.inlein/port
Restart the anthilld service
service anthilld restart
If that doesn't work, upgrade to the latest anthilld version (recommended regardless if the issue was fixed or not): upgrade-anthilld

And after that it looks like at first the antman works when I run service antman status, but after a while I get:

service antman status
* status: crashed

So then I tried to also run upgrade-anthilld, and I got the following:

upgrade-anthilld
* Installing/updating antHill ...
* Download 'anthill-daemon-1.3.2.tar.gz' failed, aborting.

So I definitely know I am now running antman 3.4.4:

antman-version
3.4.4

but I'm still stuck with the antman interface not working properly. What can I try next in the debugging?

Click for thumbs down.0Click for thumbs up.0
Last edited on September 26, 2023, 4:31 pm by alexl
alexl@alexl
4 PostsTopic Author
#6 · September 26, 2023, 5:19 pm

Is this possibly related to the edgelinux version?  It looks like I'm on 0.5 (see below - I think I was previously confused about the version number of antman, 0.5 was the edgeLinux version). Does antman 3.4.4 run on that version?  Whatever previous version of antman I had seemed to work OK.

cat /etc/edgeLinux/version
0.5.0

It also seems like I cannot connect to anthill either - "Last seen" date says "a month ago".

Click for thumbs down.0Click for thumbs up.0
Last edited on September 26, 2023, 8:15 pm by alexl