• Issues with QWK and ftp

    From Dumas Walker@VERT/CAPCITY2 to ALL on Sunday, November 17, 2019 11:18:00
    Good morning,
    I have two users (+ myself), and several QWK nodes, that use the QWK via
    ftp function of Synchronet. Over the past several days, all three users
    and at least one of the nodes has had issues with the QWK session starting
    and never completing.

    I am not seeing any errors on this end. On the user end, it varies between getting an eventual time out message to never seeing anything and
    eventually aborting (at which point they get a 421 message).

    I used to assume it was happening to me because my QWK packets were large,
    so I have started downloading them via telnet sessions. However, it
    is turning out that is not the issue as one of the users, when they
    got a packet via telnet, only had 3 messages waiting.

    Has anyone else had this happen? Like I said, the session appear normal on this end, and I don't see anything in the logs.

    Thanks!


    * SLMR 2.1a * Gimme three chili dogs and a malt.

    ---
    þ Synchronet þ CAPCITY2 * capcity2.synchro.net * Telnet/SSH:2022/Rlogin/HTTP
  • From Digital Man@VERT to Dumas Walker on Sunday, November 17, 2019 16:33:28
    Re: Issues with QWK and ftp
    By: Dumas Walker to ALL on Sun Nov 17 2019 11:18 am

    Good morning,
    I have two users (+ myself), and several QWK nodes, that use the QWK via
    ftp function of Synchronet. Over the past several days, all three users
    and at least one of the nodes has had issues with the QWK session starting and never completing.

    I am not seeing any errors on this end. On the user end, it varies between getting an eventual time out message to never seeing anything and
    eventually aborting (at which point they get a 421 message).

    I used to assume it was happening to me because my QWK packets were large, so I have started downloading them via telnet sessions. However, it
    is turning out that is not the issue as one of the users, when they
    got a packet via telnet, only had 3 messages waiting.

    Has anyone else had this happen? Like I said, the session appear normal on this end, and I don't see anything in the logs.

    Thanks!

    You'll need to examine the logs more closely. The Event Thread is responsible for creating QWK packets that are to be FTP-downloaded, so if your Event Thread isn't running or busy doing something else, QWK packets can't be created for FTP-download.

    digital man

    Synchronet/BBS Terminology Definition #8:
    BPS = Bits Per Second
    Norco, CA WX: 85.0øF, 12.0% humidity, 0 mph S wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Dumas Walker@VERT/CAPCITY2 to DIGITAL MAN on Monday, November 18, 2019 17:40:00
    You'll need to examine the logs more closely. The Event Thread is responsible f
    r creating QWK packets that are to be FTP-downloaded, so if your Event Thread i
    n't running or busy doing something else, QWK packets can't be created for FTP-
    ownload.

    Well, I was looking for error messages when there apparently are none. Turns out I found a hung binkit session (with no error messages) that was the apparent cause. Tried to "q"uit sbbs but it then "hung" waiting for the terminal server to close. After waiting 5-10min, I gave the process a
    sigkill, restarted it, let it get started, "q"uit it again (to get a clean
    exit because ftp qwk still was not working), and started it a third time.

    That seemed to clear it up.

    I find it a little odd, though, that this causes the QWK packets not to
    create but yet won't stop another binkit event from eventually running and, too, also getting stuck. Seems like QWK packets via ftp were the only
    thing not working.

    The hung binkit sessions are with a mystic system. For some reason, we are bouncing the same few messages back and forth between us, and the packet has got so big that one of our sides times-out trying to send it but does not actually tell the other side, causing binkit to just sit there like it thinks the transfer is still going on.

    Thing is, I don't even get that echo from the system in question and I cannot figure out what is causing sbbsecho to try to send it back as an OUT (like
    it is netmail) instead of sending it in an archive like it does other echo traffic. Also don't know why his system keeps sending it back to mine since
    I don't pull that echo from that node, and it seems to send it back even
    after I delete the offending OUT on this side.

    I have ZIPped up the OUT file in question and am going to share it with that sysop to see if he knows why his system keeps sending it back.

    I may send it to you, too, if you want to see it. It is a set of old
    messages from Bill McGarrity from c2017 (I think) that somehow started
    flowing back and forth between us about a month ago in a bridged newsgroup.
    I have not seen these messages show up in said newsgroup as seeing mail
    from Bill would have raised a "has he come back?" flag, or a "I have a
    dupes issue" flag. :)

    If there is an easier way to clear out a thread that is hung up like that
    I would like to know. I tried deleting the *.bsy file and that did not
    seem to clear it up.

    Thanks!


    * SLMR 2.1a * "Cool! I broke his brain!" - Bart on Principal Skinner

    ---
    þ Synchronet þ CAPCITY2 * capcity2.synchro.net * Telnet/SSH:2022/Rlogin/HTTP
  • From Digital Man@VERT to Dumas Walker on Monday, November 18, 2019 18:20:35
    Re: Issues with QWK and ftp
    By: Dumas Walker to DIGITAL MAN on Mon Nov 18 2019 05:40 pm

    You'll need to examine the logs more closely. The Event Thread is responsible f
    r creating QWK packets that are to be FTP-downloaded, so if your Event Thread i
    n't running or busy doing something else, QWK packets can't be created for FTP-
    ownload.

    Well, I was looking for error messages when there apparently are none.
    Turns out I found a hung binkit session (with no error messages) that was the apparent cause. Tried to "q"uit sbbs but it then "hung" waiting for the terminal server to close. After waiting 5-10min, I gave the process a sigkill, restarted it, let it get started, "q"uit it again (to get a clean exit because ftp qwk still was not working), and started it a third time.

    That seemed to clear it up.

    I find it a little odd, though, that this causes the QWK packets not to create but yet won't stop another binkit event from eventually running and, too, also getting stuck. Seems like QWK packets via ftp were the only
    thing not working.

    *inbound* BinkP sessions are handled via service (not timed events).

    If any non-background timed evnet hangs, it'll stop all other timed events from running, not *just* QWK packet packing/unpacking.

    The hung binkit sessions are with a mystic system. For some reason, we are bouncing the same few messages back and forth between us, and the packet has got so big that one of our sides times-out trying to send it but does not actually tell the other side, causing binkit to just sit there like it thinks the transfer is still going on.

    The log output from one of these sessions could be helpful in debugging and fixing the problem.

    Thing is, I don't even get that echo from the system in question and I cannot figure out what is causing sbbsecho to try to send it back as an OUT (like
    it is netmail) instead of sending it in an archive like it does other echo traffic.

    A careful read of the log output will likely explain what's going on.

    Also don't know why his system keeps sending it back to mine since
    I don't pull that echo from that node, and it seems to send it back even after I delete the offending OUT on this side.

    I have ZIPped up the OUT file in question and am going to share it with that sysop to see if he knows why his system keeps sending it back.

    I may send it to you, too, if you want to see it.

    Sure. But the log output is probably of more value.

    It is a set of old
    messages from Bill McGarrity from c2017 (I think) that somehow started flowing back and forth between us about a month ago in a bridged newsgroup.
    I have not seen these messages show up in said newsgroup as seeing mail from Bill would have raised a "has he come back?" flag, or a "I have a
    dupes issue" flag. :)

    If there is an easier way to clear out a thread that is hung up like that
    I would like to know. I tried deleting the *.bsy file and that did not
    seem to clear it up.

    No, it depends on "the thread". BinkIT should not get hung up in the first place, so it sounds like a bug that just needs to be fixed.

    digital man

    Synchronet/BBS Terminology Definition #33:
    FTSC = FidoNet Technical Standards Committee
    Norco, CA WX: 77.0øF, 20.0% humidity, 5 mph E wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Dumas Walker@VERT/CAPCITY2 to DIGITAL MAN on Tuesday, November 19, 2019 20:00:00
    The log output from one of these sessions could be helpful in debugging and fi
    ng the problem.
    [...]
    A careful read of the log output will likely explain what's going on.

    I looked it over pretty good a couple of times and could never find an
    error, just where the connection began and then eventually the messages
    ceasing with nothing else happened. I have all of my logging going to
    syslog. If there is nothing in there that is too sensitive,
    I would not mind sharing it. I kept copies of some days it happened.

    Actually, there is an error of sorts. The first 'evnt BINKIT' message in
    the syslog for 11/14 is at approx. 3:34am where it says "Send failure." That event would have started sometime the day before, several hours earlier and I am not able to find the previous message. The event continues and exits
    with '0' at 03:35.

    At 06:00 that same day, the next timed BINKIT event starts. The 'evnt
    BINKIT' messages soon disappear, and the "Send failure" message does not
    show up until 13:43 that afternoon. I do not see any 'evnt BINKIT'
    messages in between, although they could be there as the search function of mcview appears to be seriously challenged with a file so large. That event ends at 13:44 and then a bunch of stuff that has been waiting in the
    meantime kicks off, including another BINKIT timed event that stops sending
    any messages at 13:46. That was on 11/14. The next 'evnt BINKIT' message
    I can find in the syslog is a "Send failure" at 03:33 on *11/16*.

    Like I said I would not mind sharing but I do not see anything showing up between what appears to be a normal message and the "Send failure" several hours later. At the very least, I think it might be good to have it time-out and continue after a few minutes of inactivity.


    * SLMR 2.1a * May The Force be with you...

    ---
    þ Synchronet þ CAPCITY2 * capcity2.synchro.net * Telnet/SSH:2022/Rlogin/HTTP
  • From Digital Man@VERT to Dumas Walker on Wednesday, November 20, 2019 14:09:55
    Re: Issues with QWK and ftp
    By: Dumas Walker to DIGITAL MAN on Tue Nov 19 2019 08:00 pm

    The log output from one of these sessions could be helpful in debugging and fi
    ng the problem.
    [...]
    A careful read of the log output will likely explain what's going on.

    I looked it over pretty good a couple of times and could never find an error, just where the connection began and then eventually the messages ceasing with nothing else happened. I have all of my logging going to syslog. If there is nothing in there that is too sensitive,
    I would not mind sharing it. I kept copies of some days it happened.

    Actually, there is an error of sorts. The first 'evnt BINKIT' message in the syslog for 11/14 is at approx. 3:34am where it says "Send failure."
    That event would have started sometime the day before, several hours earlier and I am not able to find the previous message. The event continues and exits
    with '0' at 03:35.

    At 06:00 that same day, the next timed BINKIT event starts. The 'evnt BINKIT' messages soon disappear, and the "Send failure" message does not show up until 13:43 that afternoon. I do not see any 'evnt BINKIT'
    messages in between, although they could be there as the search function of mcview appears to be seriously challenged with a file so large. That event ends at 13:44 and then a bunch of stuff that has been waiting in the meantime kicks off, including another BINKIT timed event that stops sending any messages at 13:46. That was on 11/14. The next 'evnt BINKIT' message
    I can find in the syslog is a "Send failure" at 03:33 on *11/16*.

    It sounds like that "Send failure" might be exactly the log message which indicates the problem.

    Like I said I would not mind sharing but I do not see anything showing up between what appears to be a normal message and the "Send failure" several hours later. At the very least, I think it might be good to have it time-out and continue after a few minutes of inactivity.

    I agree, that's how it's supposed to work already. I look into why it might not be in your case.

    digital man

    Synchronet/BBS Terminology Definition #60:
    SCFG = Synchronet Configuration Utility
    Norco, CA WX: 55.9øF, 62.0% humidity, 3 mph WSW wind, 0.01 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Rampage@VERT/SESTAR to Dumas Walker on Thursday, November 21, 2019 19:00:38
    Re: Issues with QWK and ftp
    By: Dumas Walker to DIGITAL MAN on Tue Nov 19 2019 20:00:00

    show up until 13:43 that afternoon. I do not see any 'evnt BINKIT'
    messages in between, although they could be there as the search
    function of mcview appears to be seriously challenged with a file
    so large. That event ends at 13:44 and then a bunch of stuff that
    has been waiting in the

    grep is your friend...

    grep -e " evnt .*BINKIT" /var/log/syslog

    and you can limit it by months or days by adding the proper part to the beginning of the expression being sought... i generally use zgrep when i'm going through the logs because i archive them daily and may need to search through the gzipped ones, too...

    zgrep -E -e "^Nov 2. .* evnt .*BINKIT" /var/log/syslog

    will show all BINKIT lines for days in november from the 20th to the 29th...

    change the "2." to "[12]." and it'll be the 10th through the 29th...

    maybe that helps?


    )\/(ark

    ---
    þ Synchronet þ The SouthEast Star Mail HUB - SESTAR
  • From Dumas Walker@VERT/CAPCITY2 to DIGITAL MAN on Thursday, November 21, 2019 17:05:00
    It sounds like that "Send failure" might be exactly the log message which indic
    tes the problem.

    It does not give an error number or anything along with the message.

    hours later. At the very least, I think it might be good to have it time-out and continue after a few minutes of inactivity.
    I agree, that's how it's supposed to work already. I look into why it might not
    be in your case.

    If you need me to send the entire logs, let me know.

    This connection was with a mystic system. Here is a clip from the log at
    the beginning of a connection:

    Nov 14 06:00:25 myname synchronet: evnt BINKIT Attempting callout for 21:1/100@fsxnet, file: /sbbs/fido/out.015/00010064.out
    Nov 14 06:00:25 myname synchronet: evnt BINKIT JSBinkP/1.119 callout to 21:1/100@fsxnet started
    Nov 14 06:00:25 myname synchronet: evnt BINKIT Connecting to 21:1/100@fsxnet
    at agency.bbs.nz:24556
    Nov 14 06:00:26 myname synchronet: evnt BINKIT Peer version: Mystic/1.12A43 binkp/1.0
    Nov 14 06:00:26 myname synchronet: evnt BINKIT Authentication successful: secure
    Nov 14 06:00:26 myname synchronet: evnt BINKIT Sending file: /sbbs/fido/out.015/00010064.out (121.0KB)
    Nov 14 06:00:28 myname synchronet: evnt BINKIT Receiving file: /sbbs/temp/event/0000ffb5.th3 (77.7KB)
    [...]
    Nov 14 13:43:33 myname synchronet: evnt BINKIT Send failure
    Nov 14 13:43:33 myname synchronet: evnt BINKIT Send failure
    Nov 14 13:43:33 myname synchronet: evnt BINKIT Sent file: /sbbs/fido/out.015/00010064.out (121.0KB)
    Nov 14 13:43:33 myname synchronet: evnt BINKIT Failed to receive the whole
    file '/sbbs/temp/event/0000ffb5.th3'.
    Nov 14 13:43:33 myname synchronet: evnt BINKIT Attempting callout for [...]

    Ironically, it reports that it did send a file but failed to receive one.


    * SLMR 2.1a * It's the Bucket woman! She'll sing at me!!!

    ---
    þ Synchronet þ CAPCITY2 * capcity2.synchro.net * Telnet/SSH:2022/Rlogin/HTTP
  • From Dumas Walker@VERT/CAPCITY2 to RAMPAGE on Friday, November 22, 2019 18:37:00
    zgrep -E -e "^Nov 2. .* evnt .*BINKIT" /var/log/syslog

    will show all BINKIT lines for days in november from the 20th to the 29th...

    change the "2." to "[12]." and it'll be the 10th through the 29th...

    maybe that helps?


    Yes. One question, the syslogs that are gziped change names (i.e.
    syslog.2.gz, syslog.3.gz, etc. Does the command line above go through all
    of the syslog.* files, or do you have to change the name for each one?

    Mine roll over daily.

    Thanks!


    * SLMR 2.1a * "Einstein? Who's he? Another troublemaker?" - H.Baines

    ---
    þ Synchronet þ CAPCITY2 * capcity2.synchro.net * Telnet/SSH:2022/Rlogin/HTTP
  • From Rampage@VERT/SESTAR to Dumas Walker on Saturday, November 23, 2019 08:47:32
    Re: Issues with QWK and ftp
    By: Dumas Walker to RAMPAGE on Fri Nov 22 2019 18:37:00


    Yes. One question, the syslogs that are gziped change names (i.e. syslog.2.gz, syslog.3.gz, etc. Does the command line above go through all of the syslog.* files, or do you have to change the name for each one?

    no, something like

    zgrep -E -e " evnt .*BINKIT " /var/log/syslog*

    will go through all of them...

    zgrep -E -e " evnt .*BINKIT " /var/log/syslog.{1,2,3}

    will look only in files 1 through 3... shell file name globbing can be ""fun"" to work with... they can be especially tricky when working with syslogs where syslog.1 does not have the .gz extension but the others do... also especially since zgrep automatically adds .gz if it cannot find the specified file... i've been testing these command lines for 30+ minutes while writing this and... well... ugh...

    Mine roll over daily.

    mine, too...


    )\/(ark

    ---
    þ Synchronet þ The SouthEast Star Mail HUB - SESTAR
  • From Dumas Walker@VERT/CAPCITY2 to RAMPAGE on Saturday, November 23, 2019 16:02:00
    Yes. One question, the syslogs that are gziped change names (i.e.
    syslog.2.gz, syslog.3.gz, etc. Does the command line above go through all >DW> of the syslog.* files, or do you have to change the name for each one?

    no, something like

    zgrep -E -e " evnt .*BINKIT " /var/log/syslog*

    will go through all of them...

    zgrep -E -e " evnt .*BINKIT " /var/log/syslog.{1,2,3}

    Thanks for the pointers. I had heard of grep but never of zgrep!


    * SLMR 2.1a * Goodness! That was close! I almost gave a damn.

    ---
    þ Synchronet þ CAPCITY2 * capcity2.synchro.net * Telnet/SSH:2022/Rlogin/HTTP
  • From Rampage@VERT/SESTAR to Dumas Walker on Sunday, November 24, 2019 09:23:15
    Re: Issues with QWK and ftp
    By: Dumas Walker to RAMPAGE on Sat Nov 23 2019 16:02:00


    zgrep -E -e " evnt .*BINKIT " /var/log/syslog.{1,2,3}

    Thanks for the pointers. I had heard of grep but never of zgrep!

    zgrep comes with gzip and is simply a wrapper around grep... zgrep first unzips the archive and then fires grep at it with the provided grep command line params... that's why i always use -E with zgrep to for grep's extended regex capability... i also always use -e expressly to denote the expression but it is not really needed...


    )\/(ark

    ---
    þ Synchronet þ The SouthEast Star Mail HUB - SESTAR