• Service recycling

    From alterego@VERT/ALTERANT to All on Sunday, July 19, 2020 22:14:09
    I know this has come up before, but I'm having trouble finding the answers - if somebody could remind me, or point me to the right place.

    On 1 configuration of SBBS, which is started as root, but switches to user "bbsowner", I've noticed two things (sbbs is owned/group bbsowner as well):

    * Semaphore events are not running - ie: when fidoout.now/fidoin.now is created, the fido event doesnt run (but it does run when SBBS is first started before it switches to "bbsowner").

    * Service recycling is not happening (and that message is logged when SBBS first starts).

    (I'm thinking they may be related?)

    In sbbs.ini, in the [UNIX] section, it does have User/Group = bbsowner, and daemonize = false.

    How I do I fix this, ie: get service recycling happening when I change ini files and have semaphore events run when they are created?

    ...

    ... Heavy, adj.: Seduced by the chocolate side of the force.

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to alterego on Sunday, July 19, 2020 11:39:27
    Re: Service recycling
    By: alterego to All on Sun Jul 19 2020 10:14 pm

    I know this has come up before, but I'm having trouble finding the answers - if somebody could remind me, or point me to the right place.

    On 1 configuration of SBBS, which is started as root, but switches to user "bbsowner", I've noticed two things (sbbs is owned/group bbsowner as well):

    * Semaphore events are not running - ie: when fidoout.now/fidoin.now is created, the fido event doesnt run (but it does run when SBBS is first started before it switches to "bbsowner").

    Check permissions on the data directory. Make sure 'bbsowner' has all permissions on the directory.

    * Service recycling is not happening (and that message is logged when SBBS first starts).

    "that message" is this: "Disabling Terminal Server recycle support"? You can either define DONT_BLAME_SYNCHRONET when building sbbs if you don't care about the security implications (sbbs potentially running with root permissions) or you can bind to port numbers >= 1024.

    (I'm thinking they may be related?)

    I don't think so.

    digital man

    Sling Blade quote #21:
    Karl: Coffee makes me nervous when I drink it. Mmm.
    Norco, CA WX: 83.5F, 45.0% humidity, 10 mph E wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From Underminer@VERT/UNDRMINE to Digital Man on Sunday, July 19, 2020 16:28:11
    Re: Service recycling
    By: Digital Man to alterego on Sun Jul 19 2020 11:39 am

    * Semaphore events are not running - ie: when fidoout.now/fidoin.now
    is created, the fido event doesnt run (but it does run when SBBS is
    Check permissions on the data directory. Make sure 'bbsowner' has all permissions on the directory.
    * Service recycling is not happening (and that message is logged
    can either define DONT_BLAME_SYNCHRONET when building sbbs if you don't permissions) or you can bind to port numbers >= 1024.

    I've run into this just recently when migrating. You may want to check sbbs.ini and increase bindretrycount and/or bindretrydelay. What was happening to me with the defaults is that I'd make an update, the system would try to recycle, not be able to bind after only 15 seconds, try again twice, then hang but claim to be online. When that happened semaphore procecssing also stopped.

    Setting retrybinddelay to 60 fixed the issue. I could probably lower it, but that's a reasonable number that works for the moment.
    ---
    Underminer
    The Undermine BBS - bbs.undermine.ca:423
    Fidonet: 1:342/17
    ---
    Synchronet The Undermine - bbs.undermine.ca:423
  • From alterego@VERT/ALTERANT to Digital Man on Monday, July 20, 2020 09:35:32
    Re: Service recycling
    By: Digital Man to alterego on Sun Jul 19 2020 11:39 am

    "that message" is this: "Disabling Terminal Server recycle support"? You can either define DONT_BLAME_SYNCHRONET when building sbbs if you don't care about the security implications (sbbs potentially running with root permissions) or you can bind to port numbers >= 1024.

    Ahh, OK, so recycling only happens if SBBS runs as root, or you have the DONT_BLAME.. set? I hadnt thought about the privileged ports and they impact it has to server recycling... Got it :)

    * Semaphore events are not running - ie: when fidoout.now/fidoin.now
    is created, the fido event doesnt run (but it does run when SBBS is
    first
    started before it switches to "bbsowner").
    Check permissions on the data directory. Make sure 'bbsowner' has all permissions on the directory.

    So its all owned by bbsowner, and to be sure I stopped synchro, did a chown -R and restarted it.

    As I say, it runs the event when it starts:

    7/19 23:14:15 evnt FIDOIN Running native timed event: FIDOIN

    And then when it switches

    Successfully changed user-id to bbsowner

    After that nothing.

    bbsowner@altertex:/sbbs/data$ ls -ald . *now
    drwx------ 11 bbsowner bbsowner 4096 Jul 19 23:26 .
    -rw-r--r-- 1 bbsowner bbsowner 0 Jul 19 23:18 fidoin.now
    -rw------- 1 bbsowner bbsowner 0 Jul 19 23:26 fidoout.now bbsowner@altertex:/sbbs/data$ date
    Sun Jul 19 23:34:42 UTC 2020

    ...

    ... Leonardo da Vinci could write with one hand and draw with the other at the same time.

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to alterego on Sunday, July 19, 2020 20:52:25
    Re: Service recycling
    By: alterego to Digital Man on Mon Jul 20 2020 09:35 am

    Re: Service recycling
    By: Digital Man to alterego on Sun Jul 19 2020 11:39 am

    "that message" is this: "Disabling Terminal Server recycle support"? You can either define DONT_BLAME_SYNCHRONET when building sbbs if you don't care about the security implications (sbbs potentially running with root permissions) or you can bind to port numbers >= 1024.

    Ahh, OK, so recycling only happens if SBBS runs as root, or you have the DONT_BLAME.. set?

    Or you use the non-system ports.
    Or you have the bind capabilities on the sbbs executable: http://wiki.synchro.net/howto:linux_non-root

    I hadnt thought about the privileged ports and they impact
    it has to server recycling... Got it :)

    * Semaphore events are not running - ie: when fidoout.now/fidoin.now
    is created, the fido event doesnt run (but it does run when SBBS is
    first
    started before it switches to "bbsowner").
    Check permissions on the data directory. Make sure 'bbsowner' has all permissions on the directory.

    So its all owned by bbsowner, and to be sure I stopped synchro, did a chown -R and restarted it.

    As I say, it runs the event when it starts:

    7/19 23:14:15 evnt FIDOIN Running native timed event: FIDOIN

    And then when it switches

    Successfully changed user-id to bbsowner

    After that nothing.

    bbsowner@altertex:/sbbs/data$ ls -ald . *now
    drwx------ 11 bbsowner bbsowner 4096 Jul 19 23:26 .
    -rw-r--r-- 1 bbsowner bbsowner 0 Jul 19 23:18 fidoin.now
    -rw------- 1 bbsowner bbsowner 0 Jul 19 23:26 fidoout.now bbsowner@altertex:/sbbs/data$ date
    Sun Jul 19 23:34:42 UTC 2020

    Okay, is it possible that the event thread is busy doing something? Only one (non-background) thread can be started at a time. If the event thread runs a foreground event which hangs, it won't auto-recover and run any more events.

    digital man

    Synchronet "Real Fact" #98:
    Synchronet v3.12a was released on December 31st of 2004 (Rob's birthday). Norco, CA WX: 73.1F, 58.0% humidity, 6 mph ENE wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From alterego@VERT/ALTERANT to Digital Man on Monday, July 20, 2020 17:22:14
    Re: Service recycling
    By: Digital Man to alterego on Sun Jul 19 2020 08:52 pm

    Okay, is it possible that the event thread is busy doing something? Only one (non-background) thread can be started at a time. If the event thread runs a foreground event which hangs, it won't auto-recover and run any more events.

    Something, I'm thinking yes. What - no idea?

    I shutdown SBBS today using "q" on the console, and after waiting a loooong time for things to stop, I noticed:

    7/20 07:06:39 term Done waiting for node threads to terminate
    7/20 07:06:39 term Waiting for events thread to terminate...

    And then many lines of "Terminal Server thread still running".

    After several minutes, I killed the process (sick of waiting)...

    However, before I killed 15456, I noticed this:
    $ ps -Af|grep sbbs
    bbsowner 15429 15428 2 Jul19 pts/0 00:10:48 sbbs
    root 15456 15429 0 Jul19 pts/0 00:00:00 sbbs

    Should 15456 be root, even thought its a child of 15429? But when I killed (had to be signal 9) it I saw this:

    7/20 07:18:30 evnt FIDOIN Timed event: FIDOIN returned 0
    7/20 07:18:31 evnt BBS Events BBS Events thread terminated
    7/20 07:18:31 term Done waiting for events thread to terminate
    7/20 07:18:31 term Terminal Server thread terminating

    I think that was the original FIDOIN event that was started, before it switched to bbsowner (in my previous message)?

    (And when I re-started it, I saw FIDOIN running but no "returned".)

    ...

    ... What?!? This isn't the Files section?!?

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From alterego@VERT/ALTERANT to Digital Man on Monday, July 20, 2020 17:28:53
    Re: Service recycling
    By: alterego to Digital Man on Mon Jul 20 2020 05:22 pm

    (And when I re-started it, I saw FIDOIN running but no "returned".)

    So yup, as a follow on, when SBBS started, I had this:

    deon@altertex:~$ ps -Af|grep sbbs
    bbsowner 28470 28469 2 07:19 pts/0 00:00:05 sbbs
    bbsowner 28497 28470 0 07:19 pts/0 00:00:00 sbbs

    IE: This time 28497 is running as SBBS (so now I'm confused), but I guessed that it was still the FIDOIN process.

    So I killed it just "kill", nothing happened.

    So I killed it with signal 9, and then:

    7/20 07:22:47 evnt FIDOIN Timed event: FIDOIN returned 0
    7/20 07:22:47 evnt FIDOOUT Running native timed event: FIDOOUT
    7/20 07:22:47 evnt FIDOOUT Timed event: FIDOOUT returned 0
    <and a flurry of other events during startup>

    Now when I touch a semafore, they run (including FIDOIN).

    And hints on that first event triggered by a semafore?

    ...

    ... Chuck Norris did in fact, build Rome in a day.

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to alterego on Monday, July 20, 2020 12:42:34
    Re: Service recycling
    By: alterego to Digital Man on Mon Jul 20 2020 05:22 pm

    Re: Service recycling
    By: Digital Man to alterego on Sun Jul 19 2020 08:52 pm

    Okay, is it possible that the event thread is busy doing something? Only one (non-background) thread can be started at a time. If the event thread runs a foreground event which hangs, it won't auto-recover and run any more events.

    Something, I'm thinking yes. What - no idea?

    Your sbbs log output should tell you "what": The last thing executed by the event thread.

    I shutdown SBBS today using "q" on the console, and after waiting a loooong time for things to stop, I noticed:

    7/20 07:06:39 term Done waiting for node threads to terminate
    7/20 07:06:39 term Waiting for events thread to terminate...

    And then many lines of "Terminal Server thread still running".

    After several minutes, I killed the process (sick of waiting)...

    However, before I killed 15456, I noticed this:
    $ ps -Af|grep sbbs
    bbsowner 15429 15428 2 Jul19 pts/0 00:10:48 sbbs
    root 15456 15429 0 Jul19 pts/0 00:00:00 sbbs

    Should 15456 be root, even thought its a child of 15429?

    Possibly if it's try to rebind a port which needs root-permissions? Best to just use setcap and not need to be root.

    But when I killed
    (had to be signal 9) it I saw this:

    7/20 07:18:30 evnt FIDOIN Timed event: FIDOIN returned 0
    7/20 07:18:31 evnt BBS Events BBS Events thread terminated
    7/20 07:18:31 term Done waiting for events thread to terminate
    7/20 07:18:31 term Terminal Server thread terminating

    I think that was the original FIDOIN event that was started, before it switched to bbsowner (in my previous message)?

    (And when I re-started it, I saw FIDOIN running but no "returned".)

    That would be bad. Need to resolve that - find out what SBBSecho is doing, why it's not terminating (i.e. look at the console output of the sbbsecho process or the last lines added to sbbsecho.log).

    digital man

    Sling Blade quote #15:
    Doyle Hargraves: What'cha doin' with that lawn mower blade Karl?
    Norco, CA WX: 83.2F, 49.0% humidity, 8 mph NNE wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From Digital Man@VERT to alterego on Monday, July 20, 2020 12:43:23
    Re: Service recycling
    By: alterego to Digital Man on Mon Jul 20 2020 05:28 pm

    Re: Service recycling
    By: alterego to Digital Man on Mon Jul 20 2020 05:22 pm

    (And when I re-started it, I saw FIDOIN running but no "returned".)

    So yup, as a follow on, when SBBS started, I had this:

    deon@altertex:~$ ps -Af|grep sbbs
    bbsowner 28470 28469 2 07:19 pts/0 00:00:05 sbbs
    bbsowner 28497 28470 0 07:19 pts/0 00:00:00 sbbs

    IE: This time 28497 is running as SBBS (so now I'm confused), but I guessed that it was still the FIDOIN process.

    So I killed it just "kill", nothing happened.

    So I killed it with signal 9, and then:

    7/20 07:22:47 evnt FIDOIN Timed event: FIDOIN returned 0
    7/20 07:22:47 evnt FIDOOUT Running native timed event: FIDOOUT
    7/20 07:22:47 evnt FIDOOUT Timed event: FIDOOUT returned 0
    <and a flurry of other events during startup>

    Now when I touch a semafore, they run (including FIDOIN).

    And hints on that first event triggered by a semafore?

    Need to look at sbbsecho (that's the process being started by the FIDOIN event, or at least, should be).

    digital man

    Synchronet/BBS Terminology Definition #57:
    Phreak = Telephone system hack[er]
    Norco, CA WX: 83.2F, 49.0% humidity, 8 mph NNE wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From alterego@VERT/ALTERANT to Digital Man on Tuesday, July 21, 2020 07:30:56
    Re: Service recycling
    By: Digital Man to alterego on Mon Jul 20 2020 12:43 pm

    Need to look at sbbsecho (that's the process being started by the FIDOIN event, or at least, should be).

    So I meant to add, there was no output (for the hanging process) to sbbsecho in the log.

    Now that it is running, its complaining that areas.bbs doesnt exist:

    2020-07-19 12:18:00 SBBSecho (PID 30504) exiting with error level 0, NetMail(0 imported, 1 exported, 1 packed)
    2020-07-20 07:22:47 Could not open Area File (../data/areas.bbs): No such file or directory
    2020-07-20 07:22:47 Created NetMail (1.msg) from Deon George (516:1/2) to sysop (516:1/1), attr: 0181, subject: test
    2020-07-20 07:22:47 Packing NetMail (1.msg) from Deon George (516:1/2) to sysop (516:1/1), attr: 0181, subject: test
    2020-07-20 07:22:47 Routing NetMail (1.msg) to 516:516/0
    2020-07-20 07:22:47 SBBSecho (PID 28593) exiting with error level 0, NetMail(0 imported, 1 exported, 1 packed)

    Between 19/Jul 12:18 and 20/Jul 07:22 I had restarted SBBS multiple times and had to kill it for it to shutdown - but no log for the rogue event.

    I just tried shutting down sbbs, creating those semafores and starting it, to see if I can get it to hang on that thread again - but no, now its working fine... I've done it 3 times, and each time the FIDOIN event (which is the first one that is run,) its run AFTER the "Successfully changed user-id to bbsowner". Cant get it to do that before the change user (like it was doing last time). But I did setcap sbbs yesterday, so not sure if that is contributing? <shrug>

    ...

    ... One child is not enough, but two are far too many.

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From alterego@VERT/ALTERANT to Digital Man on Tuesday, July 21, 2020 07:34:16
    Re: Service recycling
    By: alterego to Digital Man on Tue Jul 21 2020 07:30 am

    working fine... I've done it 3 times, and each time the FIDOIN event (which is the first one that is run,) its run AFTER the "Successfully changed user-id to bbsowner". Cant get it to do that before the change user (like it was doing last time). But I did setcap sbbs yesterday, so not sure if that is contributing? <shrug>

    Oh wait, I see FIDOIN did run before the change user, and completed - and ran again after the change user. (And I know nothing came in to trigger the FIDOIN event during startup.)

    So the only difference between my problems and now, is the events had finished before the switch user. (Not sure if that is related, but I thought I'd mention it). <shrug>

    ...

    ... It is easier to write an incorrect program than understand a correct one.

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to alterego on Monday, July 20, 2020 18:48:31
    Re: Service recycling
    By: alterego to Digital Man on Tue Jul 21 2020 07:30 am

    Re: Service recycling
    By: Digital Man to alterego on Mon Jul 20 2020 12:43 pm

    Need to look at sbbsecho (that's the process being started by the FIDOIN event, or at least, should be).

    So I meant to add, there was no output (for the hanging process) to sbbsecho in the log.

    Now that it is running, its complaining that areas.bbs doesnt exist:

    Does the areas.bbs file exist?

    2020-07-19 12:18:00 SBBSecho (PID 30504) exiting with error level 0, NetMail(0 imported, 1 exported, 1 packed)
    2020-07-20 07:22:47 Could not open Area File (../data/areas.bbs): No such file or directory

    If the ../data/areas.bbs file does actually exist, then this could be a symptom of a much greater problem. Best to resolve that first.

    Between 19/Jul 12:18 and 20/Jul 07:22 I had restarted SBBS multiple times and had to kill it for it to shutdown - but no log for the rogue event.

    I just tried shutting down sbbs, creating those semafores and starting it, to see if I can get it to hang on that thread again - but no, now its working fine... I've done it 3 times, and each time the FIDOIN event (which is the first one that is run,) its run AFTER the "Successfully changed user-id to bbsowner". Cant get it to do that before the change user (like it was doing last time). But I did setcap sbbs yesterday, so not sure if that is contributing? <shrug>

    Dunno.

    digital man

    Synchronet "Real Fact" #52:
    Answers to Frequently Asked Questions: http://wiki.synchro.net/faq:index
    Norco, CA WX: 79.5F, 56.0% humidity, 14 mph E wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From Digital Man@VERT to alterego on Monday, July 20, 2020 18:53:55
    Re: Service recycling
    By: alterego to Digital Man on Tue Jul 21 2020 07:34 am

    Re: Service recycling
    By: alterego to Digital Man on Tue Jul 21 2020 07:30 am

    working fine... I've done it 3 times, and each time the FIDOIN event (which is the first one that is run,) its run AFTER the "Successfully changed user-id to bbsowner". Cant get it to do that before the change user (like it was doing last time). But I did setcap sbbs yesterday, so not sure if that is contributing? <shrug>

    Oh wait, I see FIDOIN did run before the change user, and completed - and ran again after the change user. (And I know nothing came in to trigger the FIDOIN event during startup.)

    So the only difference between my problems and now, is the events had finished before the switch user. (Not sure if that is related, but I thought I'd mention it). <shrug>

    Sure. With setcap, you don't actually need to run as root at all (and thus no "switch user" necessary). You could just start sbbs as the user you want it to run as, no special sbbs.ini setting either.

    digital man

    This Is Spinal Tap quote #39:
    Airport Security Officer: Do you have any artificial plates or limbs?
    Norco, CA WX: 79.5F, 56.0% humidity, 14 mph E wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net
  • From alterego@VERT/ALTERANT to Digital Man on Tuesday, July 21, 2020 12:46:24
    Re: Service recycling
    By: Digital Man to alterego on Mon Jul 20 2020 06:48 pm

    Does the areas.bbs file exist?

    No - not yet. Its pretty much a virgin install. (And I was testing netmail sending in a prior thread.)

    ...

    ... I call things as I see them; If I didn't see them, I make them up!

    ---
    Synchronet Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to alterego on Tuesday, July 21, 2020 15:09:53
    Re: Service recycling
    By: alterego to Digital Man on Tue Jul 21 2020 12:46 pm

    Re: Service recycling
    By: Digital Man to alterego on Mon Jul 20 2020 06:48 pm

    Does the areas.bbs file exist?

    No - not yet. Its pretty much a virgin install. (And I was testing netmail sending in a prior thread.)

    Ah, that's okay then. In fact, you can operate just fine without an area file.

    digital man

    Synchronet "Real Fact" #89:
    Rob played drums on the album "Weedpuller" available for digital purchase/stream
    Norco, CA WX: 84.9F, 47.0% humidity, 7 mph E wind, 0.00 inches rain/24hrs

    ---
    Synchronet Vertrauen Home of Synchronet [vert/cvs/bbs].synchro.net