Skip to content

lib: port rimraf to core as shutil.rmtree

Rodrigo Muino Tomonari requested to merge github/fork/iansu/shutil-rmtree into master

The @nodejs/tooling group has been working on porting rimraf into Node.js Core: https://github.com/nodejs/tooling/issues/13. We've spoken with @isaacs and this is done with his cooperation: https://github.com/iansu/node/pull/1#issuecomment-489389946. We've chosen to implement this in a new module called shutil rather than adding it to the existing fs.rmdir code. There seems to be interest in this as a similar PR was recently opened: #28171.

1. Why?

rimraf is one of the most popular third-party modules and is used in millions of projects. The Tooling Group thinks this kind of commonly used module should be included in Node.js Core and not require a third-party package.

2. Why not add a recursive option to fs.rmdir?

By adding a flag rather than a new function feature detection becomes difficult. People have requested a way to detect the recursive option of mkdir for example.

3. Why port rimraf directly?

The rimraf code contains a number of workarounds for bugs and edge cases in platforms like Windows and sunos. These are the result of many issues and bugfixes over the years and we don't want to lose this functionality.

4. Why shutil?

The name itself comes from Python. Something like rimraf is not a filesystem primitive, instead it's an algorithm that is built on top of filesystem primitives. On other platforms these kind of utilities are often grouped under a "shell" namespace, and the Tooling team can imagine several additional utilities added to this module in the future.

5. Why is this not implemented in C++?

Like the recursive mkdir operation, recursive rmdir is not atomic. Unlike recursive mkdir, the number of operations is not bounded by the directory depth; instead, it's one operation for each file and directory encountered. Recursive mkdir cannot run its operations in parallel, either; it must create directories in serial, in descending order.

Recursive rmdir, on the other hand, would benefit from removing files in parallel--userland rimraf takes this strategy to remove files quickly as libuv's fs thread pool can handle it. There's no precedent, AFAIK, for running batch operations across multiple libuv threads in Node.js core. The problem is compounded by the necessity to implement a synchronous version of the API; consider execSync's implementation.

Several attempts have been made at implementing recursive rmdir in C++, but none so far have executed these operations using multiple threads. Because of this, a serial C++ implementation is significantly slower (@boneskull benchmarked his own implementation against userland rimraf) than a parallel, JS implementation.

In addition, a C++ implementation would need to consider the workarounds for various filesystems to provide a builtin on-par with userland rimraf. So far, no C++ implementation has attempted this.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
  • add rimraf license to combined license

Merge request reports

Loading