src/sbbs3 nopen.c 1.29 1.30
Update of /cvsroot/sbbs/src/sbbs3
In directory cvs:/tmp/cvs-serv25706

Modified Files:
nopen.c
Log Message:
Changed the yeild in the fcopy() loop to a "maybe yeild", sacrificing a bit of CPU utilization to get a large file copy completed quicker.


---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/sbbs/sbbs/-/commit/914790f5931e27c30f94e204
Modified Files:
src/sbbs3/nopen.c
Log Message:
Improve file copy (fcopy()) performance by more than an order of magnitude. Using a 256KB read buffer for copying files (rather than one byte at a time). Apparently calling fread() is not the same as a bunch of calls to fgetc()
after all. Or maybe it was the many calls to fputc() being replaced with fwrite(). Or maybe it was both. Anyway, decreased the time to copy a 1GB
file from and to a Samba share over a Gb Ethernet network from 13 minutes
to less than a minute. This matters when sbbs is backing up your data/mail
base and the files are big. The mail base is locked while being backed up
and the longer it takes to back up, the longer the mail base is locked and
no mail can be received, read or sent during that time.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Hi Rob,

On 2020-09-09 02:22:17, you wrote to Git commit to sbbs/master:

https://gitlab.synchro.net/sbbs/sbbs/-/commit/914790f5931e27c30f94e204 Modified Files:
src/sbbs3/nopen.c
Log Message:
Improve file copy (fcopy()) performance by more than an order of magnitude.Using a 256KB read buffer for copying files (rather than one byte
at a time).Apparently calling fread() is not the same as a bunch of calls to fgetc()after all. Or maybe it was the many calls to fputc() being replaced withfwrite(). Or maybe it was both. Anyway, decreased the time to copy a 1GBfile from and to a Samba share over a Gb Ethernet network from 13
minutesto less than a minute. This matters when sbbs is backing up your data/mailbase and the files are big. The mail base is locked while being backed upand the longer it takes to back up, the longer the mail base is locked andno mail can be received, read or sent during that time.

Maybe this can even be improved upon:

https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.h https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.c


Bye, Wilfred.

--- FMail-lnx64 2.1.0.18-B20170815
* Origin: FMail development HQ (2:280/464)
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Re: Re: src/sbbs3/nopen.c
By: Wilfred van Velzen to Rob Swindell on Wed Sep 09 2020 12:49 pm

Hi Rob,

On 2020-09-09 02:22:17, you wrote to Git commit to sbbs/master:

https://gitlab.synchro.net/sbbs/sbbs/-/commit/914790f5931e27c30f94e204 Modified Files:
src/sbbs3/nopen.c
Log Message:
Improve file copy (fcopy()) performance by more than an order of magnitude.Using a 256KB read buffer for copying files (rather than one byte
at a time).Apparently calling fread() is not the same as a bunch of calls to fgetc()after all. Or maybe it was the many calls to fputc() being replaced withfwrite(). Or maybe it was both. Anyway, decreased the time to copy a 1GBfile from and to a Samba share over a Gb Ethernet network from 13
minutesto less than a minute. This matters when sbbs is backing up your data/mailbase and the files are big. The mail base is locked while being backed upand the longer it takes to back up, the longer the mail base is locked andno mail can be received, read or sent during that time.

Maybe this can even be improved upon:

https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.h https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.c

Um... thanks? I was getting 400+Mbps, so I'm pretty sastified with that, and it's portable code. But I may experiment with your Linux-specific code at some point.

digital man

Sling Blade quote #24:
Karl: Kaiser blade. I hit my mother upside the head with it. Mmm... Killed her. Norco, CA WX: 75.5øF, 25.0% humidity, 0 mph WSW wind, 0.00 inches rain/24hrs

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Hi Digital,

On 2020-09-09 09:09:30, you wrote to me:

Maybe this can even be improved upon:

https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.h
https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.c

Um... thanks? I was getting 400+Mbps, so I'm pretty sastified with that, and it's portable code. But I may experiment with your Linux-specific code at some point.

And on windows just use the CopyFile() OS function...

Bye, Wilfred.

--- FMail-lnx64 2.1.0.18-B20170815
* Origin: FMail development HQ (2:280/464)
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Re: Re: src/sbbs3/nopen.c
By: Wilfred van Velzen to Digital Man on Wed Sep 09 2020 07:17 pm

Hi Digital,

On 2020-09-09 09:09:30, you wrote to me:

Maybe this can even be improved upon:

https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.h
https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.c

Um... thanks? I was getting 400+Mbps, so I'm pretty sastified with that, and it's portable code. But I may experiment with your Linux-specific code at some point.

And on windows just use the CopyFile() OS function...

Now *that* was a good tip! Thanks!

digital man

Synchronet/BBS Terminology Definition #62:
SAUCE = Standard Architecture for Universal Comment Extensions (ACiD)
Norco, CA WX: 87.8øF, 24.0% humidity, 5 mph ESE wind, 0.00 inches rain/24hrs

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Hi Digital,

On 2020-09-09 13:23:52, you wrote to me:

Maybe this can even be improved upon:

https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.h
https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.c

Um... thanks? I was getting 400+Mbps, so I'm pretty sastified with
that, and it's portable code. But I may experiment with your
Linux-specific code at some point.

And on windows just use the CopyFile() OS function...

Now *that* was a good tip! Thanks!

It's in my source code, but you have to notice it. ;-)

I'm interested if you did the speed messurement again, and if you could detect a difference? ;)

Bye, Wilfred.

--- FMail-lnx64 2.1.0.18-B20170815
* Origin: FMail development HQ (2:280/464)
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Re: Re: src/sbbs3/nopen.c
By: Wilfred van Velzen to Digital Man on Wed Sep 09 2020 10:59 pm

Hi Digital,

On 2020-09-09 13:23:52, you wrote to me:

Maybe this can even be improved upon:

https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.h
https://sourceforge.net/p/fmail/code/ci/linux/tree/copyFile.c

Um... thanks? I was getting 400+Mbps, so I'm pretty sastified with
that, and it's portable code. But I may experiment with your
Linux-specific code at some point.

And on windows just use the CopyFile() OS function...

Now *that* was a good tip! Thanks!

It's in my source code, but you have to notice it. ;-)

I'm interested if you did the speed messurement again, and if you could detect a difference? ;)

I did and CopyFile() was both faster (reduced from 37 to 5 seconds) and had lower CPU utilization. Nice!

digital man

Sling Blade quote #17:
Charles Bushman: A shovel just makes too goddamned much racket.
Norco, CA WX: 81.2øF, 44.0% humidity, 12 mph NE wind, 0.00 inches rain/24hrs

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Hi Digital,

On 2020-09-09 17:48:23, you wrote to me:

I'm interested if you did the speed messurement again, and if you could
detect a difference? ;)

I did and CopyFile() was both faster (reduced from 37 to 5 seconds) and had
lower CPU utilization. Nice!

That's much bigger improvement, then I would have expected! Nice indeed!

Bye, Wilfred.

--- FMail-lnx64 2.1.0.18-B20170815
* Origin: FMail development HQ (2:280/464)
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

Re: Re: src/sbbs3/nopen.c
By: Wilfred van Velzen to Digital Man on Thu Sep 10 2020 11:07 am

Hi Digital,

On 2020-09-09 17:48:23, you wrote to me:

I'm interested if you did the speed messurement again, and if you could
detect a difference? ;)

I did and CopyFile() was both faster (reduced from 37 to 5 seconds) and had
lower CPU utilization. Nice!

That's much bigger improvement, then I would have expected! Nice indeed!

Yeah, me too. I didn't know that Win32 function existed, so nice tip. Thanks again,

digital man

Synchronet "Real Fact" #105:
Synchronet channel: http://www.youtube.com/channel/UCsQ8iXU5yvrybyoEgo__97A Norco, CA WX: 84.1øF, 27.0% humidity, 1 mph WNW wind, 0.00 inches rain/24hrs

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/1080ab0d8a0d5dca15dcbaa2
Modified Files:
src/sbbs3/nopen.c
Log Message:
Fix new function: fopenlog()

ftello() returns 0 after fnopen(..., O_APPEND). Use filelength() instead.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/b9fe6679b94e1bc4dafa6bc3
Modified Files:
src/sbbs3/nopen.c
Log Message:
Use sopen(... SH_DENYRW) for fmutex() instead of open()

O_EXCL doesn't appear to be working very reliably over Samba, so let's add
file locking to the equation.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/2a4c59134c5de68365babe59
Modified Files:
src/sbbs3/nopen.c
Log Message:
Fix for Borland C++ build

Also, don't get hostname unless we're surely going to write to the file.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/44ce881b07782f3f622d28ea
Modified Files:
src/sbbs3/nopen.c
Log Message:
We need to initialize the fd to -1 here (since we zero-it out the fmutex_t)

and might return before setting it to anything else.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/dec152b4d4ddb260255be6bf
Modified Files:
src/sbbs3/nopen.c
Log Message:
Fix non-Windows build, broken in previous commit

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/11af4bd45489d4a180fb7e6f
Modified Files:
src/sbbs3/nopen.c
Log Message:
On Windows, lock the opened mutex file to insure *nix sopen() will fail

Even though we use O_EXCL, networked file systems still sometimes allow the secondary open() to succeed. We use record locking in xpdev's sopen() implementation for *nix, so this lock will insure such opens (and locks) will now fail.

This is an attempt to address occassional error on Vertrauen:
ERROR 2 (No such file or directory) renaming *.rep to *.bad

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/commit/bde82d5b43813992d1e2bc70
Modified Files:
src/sbbs3/nopen.c
Log Message:
Use FILE_RETRY_DELAY macro for consistent retry behavior

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net