This is file tree walker module with optimized getdirents functionalities. It also exposes all costly byproduct data like Statx buffers and dfd to client code to use other \*at() APIs as desired.
Past y2k, d_type-triggered open(O_DIRECTORY) rarely fails. Even on old FSes, failed open is as fast as lstat (the only way to avoid failure). So, optimistic open is best unless you also need lstat data for other reasons, in which case lstat+selective-open is less work. This module gives both. Client code can check lst.stx_nlink != 0 to see if further stat is needed and request lstat+selective-open via the lstats parameter. (Note: Linux grew O_DIRECTORY in 1998.)
On Linux, raw getdents64 usage saves 1 fstat (by either opendir or fdopendir) per dir. BSD/AIX/.. likely allow similar. In follow sym- link mode, loop blocking always needs (dev,ino) per dir, though. An opendir- to-stat race is easily avoided with fdopendir (maybe not on Win?). This packaged recursion is also careful to use the POSIX.2008 openat API & its sibling fstatat which largely eliminates the need to deal with full paths instead of just dirent filenames.
This is a portability shim include file for when you want code to work without deprecation warnings under younger & older Nim compilers. Just include cligen/unsafeAdr at the global scope before using unsafeAddr.Procs
proc dotOrDotDot(nm: DirName): bool {.inline, ...raises: [], tags: [], forbids: [].}
proc dstats(roots: seq[string]; recurse = 0; stats = false; chase = false; xdev = false; eof0 = false) {....raises: [IOError], tags: [WriteIOEffect], forbids: [].}
- Print file depth statistics
proc find(roots: seq[string]; recurse = 0; stats = false; chase = false; xdev = false; eof0 = false; zero = false) {....raises: [IOError], tags: [WriteIOEffect], forbids: [].}
- 2.75-4.5X faster than GNU "find /usr|.."; 1.7x faster than BSD find|fd
proc ls1AU(roots: seq[string]; recurse = 1; stats = false; chase = false; xdev = false; eof0 = false) {....raises: [IOError], tags: [WriteIOEffect], forbids: [].}
-
-r0 is 1.7-2.25x faster than GNU "ls -1AUR --color=none /usr >/dev/null".
proc lss1AU(roots: seq[string]; recurse = 1; chase = false; xdev = false; eof0 = false) {....raises: [IOError], tags: [WriteIOEffect], forbids: [].}
-
-r0 is 1.45-2.0x faster than "ls -s1AUR --color=none /usr >/dev/null".
proc showLong(label: string; dir: seq[DEnt]; wrote: var bool) {.inline, ...raises: [IOError], tags: [], forbids: [].}
proc showNames(label: string; dir: seq[string]; wrote: var bool) {.inline, ...raises: [IOError], tags: [], forbids: [].}
proc wstats(roots: seq[string]) {....raises: [OSError], tags: [ReadDirEffect], forbids: [].}
- stdlib walkDirRec impl of hierarchy-unaware part of dstats -s
Templates
template forPath(root: string; maxDepth: int; lstats, follow, xdev, eof0: bool; err: File; depth, path, nmAt, ino, dt, lst, dfd, dst, did: untyped; always, preRec, postRec, recFail: untyped)
- In the primary always code body, client code sees recursion parameter depth, file parameters path, nmAt, ino, dt, lst, and recursed directory parameters dfd, dst, did. path is the full path (rooted at root) & path[nmAt..^1] is just the dirent. ino, dt, lst are metadata; the last two may be unreliable- both dt==DT_UNKNOWN or lst.stx_nlink==0 may hold. dfd, dst, did are an open file descriptor on the dir (e.g. for fchownat-like APIs), its Statx metadata (if xdev or follow), and a HashSet of (st.st_dev,stx_ino) history to block follow mode symlink loops.
template forPath(root: string; maxDepth: int; lstats, follow, xdev, eof0: bool; err: File; depth, path, nmAt, ino, dt, lst, dfd, dst, did: untyped; always, preRec, postRec: untyped)
- 3-clause reduction of forPath
template forPath(root: string; maxDepth: int; lstats, follow, xdev, eof0: bool; err: File; depth, path, nmAt, ino, dt, lst, dfd, dst, did: untyped; always, preRec: untyped)
- 2-clause reduction of forPath
template forPath(root: string; maxDepth: int; lstats, follow, xdev, eof0: bool; err: File; depth, path, nmAt, ino, dt, lst, dfd, dst, did: untyped; always: untyped)
- 1-clause reduction of forPath
template recFailDefault(context: string; path: string; err = stderr)