• bbs crashing after ecweb update

    From DaiTengu@VERT/ENSEMBLE to All on Tuesday, July 24, 2018 00:56:19
    Hi all,

    I blew away my old ecwebv4 install and started fresh with the jsexec installer on ecweb. After doing so, I noticed that my server load on my bbs machine would shoot way up, and eventually the server would start throwing "too many open files" errors.

    I managed to stop the server load from shooting up by disabling the "files" section (removing 002-files.xjs from pages), as it took a good 50 minutes to load my files section any time someone hit that page.

    However, that didn't stop my too many open files issue. It turns out something is opening user.dat and not closing it in sbbs/http as the following lsof snippet shows:
    sbbs/http 26362 26424 root 70r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 72r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 76r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 77r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 78r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 80r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    -=-=-=-=-=-=-

    Eventually there are so many instances of user.dat open that the server hits its max open files limit.

    Has anyone run into this?

    I've had to roll back to the default synchronet web interface for the time being.

    DaiTengu

    ... Computers run on faith, not electrons.

    ---
    þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Digital Man@VERT to DaiTengu on Tuesday, July 24, 2018 00:20:39
    Re: bbs crashing after ecweb update
    By: DaiTengu to All on Tue Jul 24 2018 12:56 am

    Hi all,

    I blew away my old ecwebv4 install and started fresh with the jsexec installer on ecweb. After doing so, I noticed that my server load on my bbs machine would shoot way up, and eventually the server would start throwing "too many open files" errors.

    I managed to stop the server load from shooting up by disabling the "files" section (removing 002-files.xjs from pages), as it took a good 50 minutes to load my files section any time someone hit that page.

    However, that didn't stop my too many open files issue. It turns out something is opening user.dat and not closing it in sbbs/http as the following lsof snippet shows:
    sbbs/http 26362 26424 root 70r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 72r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 76r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 77r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 78r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    sbbs/http 26362 26424 root 80r REG 8,0 138444 5780571 /sbbs/data/user/user.dat
    -=-=-=-=-=-=-

    Eventually there are so many instances of user.dat open that the server hits its max open files limit.

    Has anyone run into this?

    I've had to roll back to the default synchronet web interface for the time being.

    The problem has been reported by before now, no one has been able to say *what* files were open by sbbs at the time of failure.

    I've forwarded the report on echicken via irc. Hopefully it's an easy fix (likely so).

    digital man

    This Is Spinal Tap quote #41:
    Ian Faith: It say's "Memphis show cancelled due to lack of advertising funds." Norco, CA WX: 77.4øF, 58.0% humidity, 0 mph S wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Al@VERT/TRMB to DaiTengu on Tuesday, July 24, 2018 02:04:45
    Re: bbs crashing after ecweb update
    By: DaiTengu to All on Tue Jul 24 2018 12:56 am

    Eventually there are so many instances of user.dat open that the server hits its max open files limit.

    Has anyone run into this?

    I've seen the "too many open files" problem but wasn't sure why that was happening.

    Looks like you shed some light on it so that's a good thing.. :)

    Ttyl :-),
    Al


    ... The large print giveth and the small print taketh away.

    ---
    þ Synchronet þ The Rusty MailBox - Penticton, BC Canada
  • From DaiTengu@VERT/ENSEMBLE to Digital Man on Tuesday, July 24, 2018 08:32:35
    Re: bbs crashing after ecweb update
    By: Digital Man to DaiTengu on Tue Jul 24 2018 12:20 am

    The problem has been reported by before now, no one has been able to say *what* files were open by sbbs at the time of failure.

    I've forwarded the report on echicken via irc. Hopefully it's an easy fix (likely so).

    I do remember seeing a few messages about too many open files in the past, but didn't know it was ecweb related, as I only caught the tail end of the conversation at the time.

    Guess I'll fire up my IRC client that's been collecting dust, maybe I can help getting it figured out :) Thanks!

    DaiTengu

    ... In the long run, we are all dead.

    ---
    þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com
  • From DaiTengu@VERT/ENSEMBLE to Al on Tuesday, July 24, 2018 08:40:41
    Re: bbs crashing after ecweb update
    By: Al to DaiTengu on Tue Jul 24 2018 02:04 am

    server hits its max open files limit.

    Has anyone run into this?

    I've seen the "too many open files" problem but wasn't sure why that was happening.

    Looks like you shed some light on it so that's a good thing.. :)

    20+ years as a linux sysadmin, my knee-jerk reaction is to start sorting 'lsof' output to figure out where the problem is. Hopefully I helped figure out how to solve the issue :)

    DaiTengu

    ... I installed a skylight in my apartment... the people who live above me are furious!

    ---
    þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com
  • From Kirkman@VERT/GUARDIAN to Digital Man on Tuesday, August 21, 2018 09:39:18
    Re: bbs crashing after ecweb update
    By: Digital Man to DaiTengu on Tue Jul 24 2018 12:20 am

    Eventually there are so many instances of user.dat open that the
    server hits its max open files limit.

    Has anyone run into this?

    I've had to roll back to the default synchronet web interface for the
    time being.

    The problem has been reported by before now, no one has been able to say *what* files were open by sbbs at the time of failure.

    I've forwarded the report on echicken via irc. Hopefully it's an easy fix (likely so).

    I also started running into that problem recently. My logs were filling up with "error 24" and "Too many open files". The lsof command clearly showed seemingly infinite references to /sbbs/data/user/user.dat.

    Anyway, I guess the problem was that I had stuck with an older version of ecwebv4 because I thought I heard that ftelnet support was being dropped.

    In any case, it had gotten really bad in the past few days. But after upgrading ecweb and sbbs, I think there's been a marked improvement.

    The real test will be the next few days to make sure the board stays up.

    --Josh

    ////--------------------------------------------------
    BiC -=- http://breakintochat.com -=- bbs wiki and blog

    ---
    þ Synchronet
  • From echicken@VERT/ECBBS to Kirkman on Tuesday, August 21, 2018 11:58:24
    Re: bbs crashing after ecweb update
    By: Kirkman to Digital Man on Tue Aug 21 2018 09:39:18

    In any case, it had gotten really bad in the past few days. But after upgrading ecweb and sbbs, I think there's been a marked improvement.

    I'm unclear on where the problem actually came from. I'm on an older build of Synchronet, so I never experienced it firsthand. I adjusted every place a new User object was created and hoped for the best. There was really only one place (userlist page) that was consistent with steps required to cause the problem, especially at the scale people were describing.

    I assume User objects etc. are destroyed after the script finishes (the HTTP request has been served), so I don't know why those files would remain open for a long time. Could be they are only open briefly, but there are *lots* of requests being served. Or maybe bad bots are holding their connections open somehow and that's preventing some cleanup? I dunno.

    Point being that it may be fixed, but I'm not satisfied as to how or why. I'll update soon and see if I can force this issue to occur on my system, which might reveal something.

    ---
    echicken
    electronic chicken bbs - bbs.electronicchicken.com - 416-273-7230
    þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com
  • From Nightfox@VERT/DIGDIST to Kirkman on Tuesday, August 21, 2018 09:42:41
    Re: bbs crashing after ecweb update
    By: Kirkman to Digital Man on Tue Aug 21 2018 09:39 am

    I also started running into that problem recently. My logs were filling up with "error 24" and "Too many open files". The lsof command clearly showed seemingly infinite references to /sbbs/data/user/user.dat.

    Anyway, I guess the problem was that I had stuck with an older version of ecwebv4 because I thought I heard that ftelnet support was being dropped.

    In any case, it had gotten really bad in the past few days. But after upgrading ecweb and sbbs, I think there's been a marked improvement.

    The real test will be the next few days to make sure the board stays up.

    For me, I kept seeing the "too many open files" issue even after upgrading ecwebv4. Digital Man put in a fix recently that fixed the issue though, so I'd try updating to newer binaries if you haven't done that recently.

    Nightfox

    ---
    þ Synchronet þ Digital Distortion: digitaldistortionbbs.com
  • From Ragnarok@VERT/DOCKSUD to Nightfox on Friday, August 24, 2018 11:42:50
    El 21/08/18 a las 13:42, Nightfox escribió:

    For me, I kept seeing the "too many open files" issue even after upgrading ecwebv4. Digital Man put in a fix recently that fixed the issue though, so I'd
    try updating to newer binaries if you haven't done that recently.


    I have "too many open files" error on my debian gnu/linux sometimes.. I
    run another services in same server with synchronet. But never see that
    issue before.
    Sometimes i need to restart sbbs daemon.
    I still have not had the opportunity to better debug to found the origin
    of this issue, but i think that some bbs process do not correctly close
    some file handle or someelse.

    ---
    þ Synchronet þ Dock Sud BBS TLD 24 HS - http://bbs.docksud.com.ar - telnet://bbs.docksud.com.ar
  • From Digital Man@VERT to Ragnarok on Friday, August 24, 2018 13:02:14
    Re: Re: bbs crashing after ecweb update
    By: Ragnarok to Nightfox on Fri Aug 24 2018 11:42 am

    El 21/08/18 a las 13:42, Nightfox escribió:

    For me, I kept seeing the "too many open files" issue even after upgrading ecwebv4. Digital Man put in a fix recently that fixed the issue though, so I'd
    try updating to newer binaries if you haven't done that recently.


    I have "too many open files" error on my debian gnu/linux sometimes.. I
    run another services in same server with synchronet. But never see that issue before.
    Sometimes i need to restart sbbs daemon.
    I still have not had the opportunity to better debug to found the origin
    of this issue, but i think that some bbs process do not correctly close
    some file handle or someelse.

    There was an issue with the JS User object which could trigger that - that issue was introduced to CVS in June (as part of a performance improvement) and fixed in August.

    digital man

    Synchronet "Real Fact" #78:
    Synchronet Match Maker had at one time over 4000 profiles of men and women. Norco, CA WX: 84.1øF, 51.0% humidity, 9 mph ENE wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Ragnarok@VERT/DOCKSUD to Digital Man on Monday, August 27, 2018 13:04:02
    El 24/08/18 a las 17:02, Digital Man escribió:
    Re: Re: bbs crashing after ecweb update
    By: Ragnarok to Nightfox on Fri Aug 24 2018 11:42 am

    El 21/08/18 a las 13:42, Nightfox escribió:

    For me, I kept seeing the "too many open files" issue even after upgrading
    ecwebv4. Digital Man put in a fix recently that fixed the issue though, so I'd
    try updating to newer binaries if you haven't done that recently.


    I have "too many open files" error on my debian gnu/linux sometimes.. I run another services in same server with synchronet. But never see that issue before.
    Sometimes i need to restart sbbs daemon.
    I still have not had the opportunity to better debug to found the origin of this issue, but i think that some bbs process do not correctly close some file handle or someelse.

    There was an issue with the JS User object which could trigger that - that issue was introduced to CVS in June (as part of a performance improvement) and
    fixed in August.


    Good to know!
    I do not see this issue lately, but was very random.

    ---
    þ Synchronet þ Dock Sud BBS TLD 24 HS - http://bbs.docksud.com.ar - telnet://bbs.docksud.com.ar