• Imported messages being modified

    From Alterego@VERT/ALTERANT to Digital Man on Tuesday, October 15, 2019 10:20:13
    Hey DM,

    I've set up a second SBBS system, while I test some messages going back and forth between it and my main SBBS. I'm playing with gpg signed messages (I'm working on something).

    SBBS "A" is my main system, SBBS "B" is the test.

    When I generate a message on "B", that gets posted into a sub using msgbase.save_msg(), sbbsexports it (BAU) sends it to "A".

    When I look at the message in "B" using msgbase.get_msg_body() - I see the format of the message is as it was when I generated it.

    When I look at the packet that sbbsecho exports from "B" to send to "A" - the message is "as is" it was when it went into the sub.

    When I look at the packet after it arrives on "A" - no change. sbbsecho tosses the packet fine. The message is in the same format still (inside the packet).

    When I look at the message in "A" using msgbase.get_msg_body() - all the 0x0A have been stripped. So the message on "A" is not the same as the message on "B". Why would the 0x0A be stripped, and can I stop that?

    Here is the hexdump from the packet:

    ../fido/inbound/badtic/5da46adb.pkt: 3044 bytes
    E7 03 01 00 E3 07 09 00 - 0E 00 17 00 20 00 1B 00 ............ ...
    00 00 02 00 E7 03 01 00 - FF 03 00 00 00 00 00 00 ................
    ...
    33 2E 30 0D 01 43 48 52 - 53 3A 20 41 53 43 49 49 3.0..CHRS: ASCII
    20 31 0D 2D 2D 2D 2D 2D - 42 45 47 49 4E 20 50 47 1.-----BEGIN PG
    50 20 4D 45 53 53 41 47 - 45 2D 2D 2D 2D 2D 0A 0A P MESSAGE-----..
    6F 77 47 31 56 6E 2B 4D - 32 39 51 64 76 34 50 72 owG1Vn+M29Qdv4Pr


    Here is the hexdump after get_msg_body():

    2D 2D 2D 2D 2D 42 45 47 - 49 4E 20 50 47 50 20 4D -----BEGIN PGP M,
    45 53 53 41 47 45 2D 2D - 2D 2D 2D 6F 77 47 31 56 ESSAGE-----owG1V,
    6E 2B 4D 32 39 51 64 76 - 34 50 72 31 74 49 68 32 n+M29Qdv4Pr1tIh2,
    4A 44 6F 55 46 63 4E 30 - 71 30 44 65 6E 63 34 64 JDoUFcN0q0Denc4d,
    ...ëîå*

    ... Waiter, this chicken's rubbery! Oh, fank you velly much! More fly lice?

    ---
    þ Synchronet þ Alterant | an SBBS in Docker on Pi!
  • From echicken@VERT/ECBBS to Alterego on Monday, October 14, 2019 21:53:14
    Re: Imported messages being modified
    By: Alterego to Digital Man on Tue Oct 15 2019 10:20:13

    When I look at the packet after it arrives on "A" - no change. sbbsecho tosses the packet fine. The message is in the same format still (inside the packet).

    When I look at the message in "A" using msgbase.get_msg_body() - all the 0x0A have been stripped. So the message on "A" is not the same as the message on "B". Why would the 0x0A be stripped, and can I stop that?

    Any chance you have "Strip outgoing line feeds" enabled in echocfg?

    ---
    echicken
    electronic chicken bbs - bbs.electronicchicken.com
    þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com
  • From Alterego@VERT/ALTERANT to echicken on Tuesday, October 15, 2019 13:29:12
    Re: Imported messages being modified
    By: echicken to Alterego on Mon Oct 14 2019 09:53 pm

    Any chance you have "Strip outgoing line feeds" enabled in echocfg?

    No - because the outgoing packet has the line feeds.

    I checked both "A" and "B" - and for Strip Outgoing Line Feeds and Strip Incoming Software CRs - both answers are No on both systems ...

    Another idea?
    ...ëîå*

    ... It works better if you plug it in.

    ---
    þ Synchronet þ Alterant | an SBBS in Docker on Pi!
  • From echicken@VERT/ECBBS to Alterego on Tuesday, October 15, 2019 00:11:57
    Re: Imported messages being modified
    By: Alterego to echicken on Tue Oct 15 2019 13:29:12

    No - because the outgoing packet has the line feeds.

    Oh, that's right - you did indicate that.

    Another idea?

    Not offhand. I would guess it's something happening on import or to do with message base settings. A quick test here makes me doubt if it's get_msg_body's fault, for whatever that's worth.

    ---
    echicken
    electronic chicken bbs - bbs.electronicchicken.com
    þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com
  • From Digital Man@VERT to Alterego on Monday, October 14, 2019 23:56:13
    Re: Imported messages being modified
    By: Alterego to Digital Man on Tue Oct 15 2019 10:20 am

    Hey DM,

    I've set up a second SBBS system, while I test some messages going back and forth between it and my main SBBS. I'm playing with gpg signed messages (I'm working on something).

    SBBS "A" is my main system, SBBS "B" is the test.

    When I generate a message on "B", that gets posted into a sub using msgbase.save_msg(), sbbsexports it (BAU) sends it to "A".

    When I look at the message in "B" using msgbase.get_msg_body() - I see the format of the message is as it was when I generated it.

    When I look at the packet that sbbsecho exports from "B" to send to "A" - the message is "as is" it was when it went into the sub.

    When I look at the packet after it arrives on "A" - no change. sbbsecho tosses the packet fine. The message is in the same format still (inside the packet).

    When I look at the message in "A" using msgbase.get_msg_body() - all the 0x0A have been stripped. So the message on "A" is not the same as the message on "B". Why would the 0x0A be stripped, and can I stop that?

    Because those are line-feeds and the FTSC specs say that line-feeds should be "ignored" (with no real definition of what "ignore" means).

    Still, SBBSecho has an option to strip line-feeds and when not enabled, they should be preserved - at least when SBBSecho is in use. But other tossers could certainly strip them between systems. It's best to not assume that all white-space characters are going to be preserved between FTN endpoints.

    digital man

    This Is Spinal Tap quote #21:
    So when you're playing you feel like a preserved moose on stage?
    Norco, CA WX: 59.1øF, 86.0% humidity, 0 mph SSW wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Alterego@VERT/ALTERANT to Digital Man on Tuesday, October 15, 2019 19:52:38
    Re: Imported messages being modified
    By: Digital Man to Alterego on Mon Oct 14 2019 11:56 pm

    Because those are line-feeds and the FTSC specs say that line-feeds should be "ignored" (with no real definition of what "ignore" means).

    Hmm... If that were true, it doesnt make sense. The message body is changed, and I would have thought that the message body not be changed between systems. (Especially in this case, when the two systems are the same software on the same OS).

    Still, SBBSecho has an option to strip line-feeds and when not enabled, they should be preserved - at least when SBBSecho is in use. But other tossers could certainly strip them between systems. It's best
    to not assume that all white-space characters are going to be preserved between FTN endpoints.

    There are no other tossers in use here - its SBBS to SBBS. So is this a bug? There is a parameter Strip Incoming Soft CRs which is set to "No" on the importing system, so I would expect that sbbsecho import the message verbatim?

    If it doesnt work that way, can it be changed so that it does?
    ...ëîå*

    ... Love is a long term investment, not a quick return loan!

    ---
    þ Synchronet þ Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to Alterego on Tuesday, October 15, 2019 10:14:52
    Re: Imported messages being modified
    By: Alterego to Digital Man on Tue Oct 15 2019 07:52 pm

    Re: Imported messages being modified
    By: Digital Man to Alterego on Mon Oct 14 2019 11:56 pm

    Because those are line-feeds and the FTSC specs say that line-feeds should be "ignored" (with no real definition of what "ignore" means).

    Hmm... If that were true, it doesnt make sense.

    It's true that 0x0A is a line-feed.

    It's true that the FidoNet specification state that line-feeds should be "ignored".

    The message body is changed,
    and I would have thought that the message body not be changed between systems. (Especially in this case, when the two systems are the same software on the same OS).

    Unfortunately, that's not always true. Different message networking protocols have different rules and limitations, so it's not true that message bodies are "not changed" between systems in all cases.

    Still, SBBSecho has an option to strip line-feeds and when not enabled, they should be preserved - at least when SBBSecho is in use. But other tossers could certainly strip them between systems. It's best
    to not assume that all white-space characters are going to be preserved between FTN endpoints.

    There are no other tossers in use here - its SBBS to SBBS. So is this a bug?

    I don't think so. SBBSecho currently *ignores* line-feeds characters from imported messages, per FTS-1 (sbbsecho.c lines 3486-87).

    There is a parameter Strip Incoming Soft CRs which is set to "No" on the importing system, so I would expect that sbbsecho import the message verbatim?

    Soft-CRs are 0x8D, so unrelated to the behavior you're talking about.

    If it doesnt work that way, can it be changed so that it does?

    Line/paragraph-endings are represented in FTN messages with a carriage-return (0x0D) character. Is it possible that your originally posted message didn't contain any carriage-returns? How was the message created exactly? Normally, message body text in SMB contains lines/paragraphs that are terminated with carriage-return/line-feed pairs (CRLF, "\r\n", 0x0D 0x0A). It could be simply that you're missing the carriage-returns from your original message text.

    digital man

    This Is Spinal Tap quote #28:
    We've got Armadillos in our trousers. It's really quite frightening.
    Norco, CA WX: 72.8øF, 58.0% humidity, 0 mph W wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Alterego@VERT/ALTERANT to Digital Man on Wednesday, October 16, 2019 10:42:17
    Re: Imported messages being modified
    By: Digital Man to Alterego on Tue Oct 15 2019 10:14 am

    Because those are line-feeds and the FTSC specs say that line-feeds should be "ignored" (with no real definition of what "ignore" means).

    "Ignored" or "removed" - I agree it looks to be a vaguely written spec, IE: it could be interpretted both ways.

    It says "All linefeeds, 0AH, should be ignored. Systems which display message text should wrap long lines to suit their application."

    I would think its appropriate to intepret this as - ignore 0x0a in input (not remove it). And when you want to wrap long lines, wrap them how you want regardless of the presence of 0x0a (when you want to display it to a user). Thus, if you store it raw, and another process pulls it out of the message store - give it to that process raw, so that it can format it how it likes.

    Still, SBBSecho has an option to strip line-feeds and when not enabled, they should be preserved - at least when SBBSecho is in use.

    So SBBSecho can be configured to not "strip" line-feeds? How?

    to not assume that all white-space characters are going to be preserved between FTN endpoints.

    OK. But "in transit", it shouldnt be modified at all right?

    I don't think so. SBBSecho currently *ignores* line-feeds characters from imported messages, per FTS-1 (sbbsecho.c lines 3486-87).

    Err... SBBSecho currently *removes* line-feeds. :)

    Line/paragraph-endings are represented in FTN messages with a carriage-return (0x0D) character. Is it possible that your originally posted message
    didn't contain any carriage-returns? How was the message created exactly?

    Yeah, thats my problem here - I'm sucking in the output from a unix app (gpg) - and thus line termination is 0x0a not 0x0d 0x0a.

    My short term fix is to base64 encode the output before posting it to the message base :)
    ...ëîå*

    ... There is no such thing as a nonracial society in a multiracial country.

    ---
    þ Synchronet þ Alterant | an SBBS in Docker on Pi!
  • From Digital Man@VERT to Alterego on Tuesday, October 15, 2019 22:13:49
    Re: Imported messages being modified
    By: Alterego to Digital Man on Wed Oct 16 2019 10:42 am

    Re: Imported messages being modified
    By: Digital Man to Alterego on Tue Oct 15 2019 10:14 am

    Because those are line-feeds and the FTSC specs say that line-feeds should be "ignored" (with no real definition of what "ignore" means).

    "Ignored" or "removed" - I agree it looks to be a vaguely written spec, IE: it could be interpretted both ways.

    It says "All linefeeds, 0AH, should be ignored. Systems which display message text should wrap long lines to suit their application."

    I would think its appropriate to intepret this as - ignore 0x0a in input (not remove it). And when you want to wrap long lines, wrap them how you want regardless of the presence of 0x0a (when you want to display it to a user). Thus, if you store it raw, and another process pulls it out of the message store - give it to that process raw, so that it can format it how it likes.

    Still, SBBSecho has an option to strip line-feeds and when not enabled, they should be preserved - at least when SBBSecho is in use.

    So SBBSecho can be configured to not "strip" line-feeds? How?

    The "don't strip line-feeds" option is with regards to exporting messages from SMB to FTN.

    to not assume that all white-space characters are going to be preserved between FTN endpoints.

    OK. But "in transit", it shouldnt be modified at all right?

    I don't have an opinion on what is "right" or wrong. It's FidoNet. It is is what it is. Other network transports have their own limitations and issues too.

    I don't think so. SBBSecho currently *ignores* line-feeds characters from imported messages, per FTS-1 (sbbsecho.c lines 3486-87).

    Err... SBBSecho currently *removes* line-feeds. :)

    Bare line-feeds, yes. There's no other way for Synchronet to "ignore" them once they're in SMB.

    Line/paragraph-endings are represented in FTN messages with a carriage-return (0x0D) character. Is it possible that your originally posted message
    didn't contain any carriage-returns? How was the message created exactly?

    Yeah, thats my problem here - I'm sucking in the output from a unix app (gpg) - and thus line termination is 0x0a not 0x0d 0x0a.

    So the fix is to terminate the lines with CRLF - see exec/postmsg.js for an example if you need one. SMB expects CRLF-terminated paragraphs/lines.

    My short term fix is to base64 encode the output before posting it to the message base :)

    That'll work too.

    digital man

    This Is Spinal Tap quote #1:
    Nigel Tufnel: These go to eleven.
    Norco, CA WX: 73.8øF, 32.0% humidity, 0 mph SW wind, 0.00 inches rain/24hrs

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Tony Langdon@VERT to Digital Man on Wednesday, October 16, 2019 20:29:00
    On 10-15-19 22:13, Digital Man wrote to Alterego <=-

    Yeah, thats my problem here - I'm sucking in the output from a unix app (gpg) - and thus line termination is 0x0a not 0x0d 0x0a.

    So the fix is to terminate the lines with CRLF - see exec/postmsg.js
    for an example if you need one. SMB expects CRLF-terminated paragraphs/lines.

    Looks like a job for unix2dos. :)


    ... I am Procrastitron. I will destroy you, eventually.
    === MultiMail/Win v0.51
    --- SBBSecho 3.03-Linux
    * Origin: Freeway BBS Bendigo,Australia freeway.apana.org.au (3:633/410)
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net