Roadmap & Changelog for rTorrent 0.11.0

[stickz]

@rakshasa Could we make a roadmap to tag a new version of rTorrent? I'm working with my partners to get the tinyxml2 changes that @kannibalox made tested and deployed to thousands of clients. We just need a release to make that happen.

My recommendation for a roadmap is:

rTorrent:

1. Test and merge tinyxml2: Resolve LTO issues #1347
2. Test and merge tinyxml2: Correctly handle commands that are flagged as not using targets #1348

libtorrent:

1. Test and merge convert structs with single operator() to funcs libtorrent#281

Recommended Changelog Includes:

libtorrent

Stability Improvements

1. Resolved a crash with Handshake buffer resizing.

   • Fix handshake buffer resizing libtorrent#254 by @rakshasa
   • fix buffer position tracking in read_port libtorrent#264 by @slingamn

2. Respect failure intervals for trackers.

   • Kannibalox feature/respect fail interval libtorrent#260 by @rakshasa and @kannibalox

3. Fix file truncation on resume with fallocate enabled

   • Updated: Fix file truncation on resume with fallocate enabled libtorrent#261 by @rakshasa and @jmakovicka

4. Change _sync* to std::atomic for cross platform compatibility.

   • Change __sync_* to std::atomic and remove unneeded code libtorrent#266 by @rakshasa
   • Fix/atomic types 2 libtorrent#267 by @rakshasa

5. Resolve Wclass-memaccess compile warning with std::memset.

   • fix -Wclass-memaccess warning libtorrent#280 by @neheb

Performance Improvements

1. Only write uncertain_pieces.timestamp when necessary.

   • Only write uncertain_pieces.timestamp when necessary libtorrent#257 by @kannibalox

2. Use emplace instead of push for std::array objects.

   • use emplace and emplace_back libtorrent#279 by @neheb

Code Cleanup

1. Convert more of functional.h to standard library constructs libtorrent#263 by @kannibalox
2. Remove unused code and convert some functions to lamdas/std::fun libtorrent#262 by @kannibalox
3. Convert rak functional methods to lambdas libtorrent#273 by @kannibalox
4. Clean up rak::functional's last methods libtorrent#274 by @kannibalox
5. remove most ptr_fun libtorrent#277 by @kannibalox
6. remove most ptr_fun libtorrent#277 by @neheb
7. Removed cppunit ldflags from common flags. libtorrent#282 by @rakshasa
8. convert structs with single operator() to funcs libtorrent#281 by @neheb

rTorrent

Stability Improvements

1. Resolved scgi software crash with SIGPIPE exception.

   • Resolve scgi software crash #1310 by @stickz
   • scgi: Fix Apple and Solaris compatibility #1329 by @stickz

2. Various build stability improvements.

   • Fix calls ar directly #1317 by @PPN-SD

3. Change _sync* to std::atomic for cross platform compatibility.

   • Convert __sync* built-ins to std::atomic #1324 by @kannibalox

4. Resolved a crash with the curl stack during shutdown.

   • Fixed curl stack shutdown with active downloads. #1338 by @rakshasa

5. Resolve Wclass-memaccess compile warning with std::memset.

   • Use std::array in partial_queue. #1344 by @rakshasa

6. Fixed d.group.name returning the wrong value.

   • fix: d.group.name should return group name instead of group index #1326 by @trim21

7. Fixed compatibility checks for file types.

   • Kannibalox fix/check file target #1327 by @kannibalox and @rakshasa

Tinyxml2 to replace xmlrpc-c

• Kannibalox feature/embedded xmlrpc #1322 by @kannibalox and @rakshasa
• xmlrpc-tinyxml2 fixes from testing against flood #1331 by @kannibalox
• tinyxml2: Correctly handle commands that are flagged as not using targets #1348 by @kannibalox
• tinyxml2: Resolve LTO issues #1347 by @stickz

Please configure rTorrent with --with-xmlrpc-tinyxml2 to enable tinyxml2. It contains 2-3 times performance improvements for small responses and up to 30 times performance improvements for large responses. This is a significant reduction in overhead!

xmlrpc-c:

Benchmark               Time             CPU   Iterations
---------------------------------------------------------
small_response       3417 ns         3413 ns       204332
large_response   46159260 ns     46074301 ns           15

tinyxml2:

Benchmark               Time             CPU   Iterations
---------------------------------------------------------
small_response       1595 ns         1593 ns       462388
large_response    1512614 ns      1509299 ns          463

Code Cleanup

1. Convert CommandMap to use std::string instead of char* #1311 by @kannibalox
2. Add clang-format/clang-tidy config files #1312 by @kannibalox
3. Removed unused code. #1314 by @rakshasa
4. Replaced depreacted rak/functional calls. #1340 by @rakshasa
5. Remove/replace functional_fun.h functions #1343 by @kannibalox
6. Removed cppunit ldflags from common flags. #1345 by @rakshasa

[trim21]

why we keep xmlrpc-c anyway?

[stickz]

why we keep xmlrpc-c anyway?

We need to test tinyxml2 first. I just found 2 problems with it today. 2 pull requests are pending to resolve them. This is a massive change. I'm working on deploying it to thousands of clients on crazy max docker. We need to test test test before we drop it.

[trim21]

also why 0.12.0 instead of 0.11? current version is 0.10, any reason skip 0.11?

[stickz]

also why 0.12.0 instead of 0.11? current version is 0.10, any reason skip 0.11?

You're right! @rakshasa likes to count by 1's for the second number. I've updated the title. Thank you.

[slingamn]

Why do you need an upstream release in order to fully test the changes? It seems backwards --- first you should deploy your release candidate, and then if your testing is successful, that would justify an upstream release :-)

[trim21]

I guess because they are clients so they want open source users test patches first

[stickz]

Why do you need an upstream release in order to fully test the changes? It seems backwards --- first you should deploy your release candidate, and then if your testing is successful, that would justify an upstream release :-)

Nobody is going to deploy a release candidate. There is no harm done with an upstream release. We can just post a hotfix afterwards if required. We've already done development testing as required. I found 2 bugs with tinyxml2 today. :-)

[neheb]

since I got pinged, anyone know what's wrong here? rakshasa/libtorrent#283

[stickz]

since I got pinged, anyone know what's wrong here? rakshasa/libtorrent#283

I would recommend breaking that down into smaller pull requests, to more effectively find problems.

We'll need your help with rakshasa/libtorrent#281 if rakshasa approves this roadmap to an upstream release. This is a very good time to hit the pause button and ship the work we've done to rTorrent to thousands of users.

[rakshasa]

Seems reasonable, I'll check out the tinyxml2 related PRs.

[rakshasa]

also why 0.12.0 instead of 0.11? current version is 0.10, any reason skip 0.11?

You're right! @rakshasa likes to count by 1's for the second number. I've updated the title. Thank you.

Just to mess things up I sync'ed both versions to 0.15.0.

[eli-schwartz]

Why do you need an upstream release in order to fully test the changes? It seems backwards --- first you should deploy your release candidate, and then if your testing is successful, that would justify an upstream release :-)

Both release candidates and final releases are useful for testing changes. It's not an "either or" thing. Usually release candidates are only used by people who manually opt in on a person to person basis though. It's absolutely useless to expect release candidates to be automatically rolled out to "thousands of clients" automatically because no one is going to ask, or expect, thousands of people in a specific group to all test something that is explicitly an experimental build.

But e.g. Gentoo would most likely package a release candidate if the package maintainer is made aware of it. And at least a few people would manually enable that release candidate, while also expecting it's their self-elected job to report obscure bugs when they do.

Nonetheless, this is NOT the same as full testing. It is standard operating procedure for software making significant changes (framework, build system, etc) that need testing, to provide at least one release cycle where both options are available, so that people can test the changes, and disable them if issues are encountered, without downgrading the software to the previous major.minor release. This is important for the project itself, too, since if people are afraid to upgrade to the new release because of one deal breaker regression then they will not test the other changes and you end up only finding and fixing a single bug per release cycle, which sucks.

Ultimately it's a balance between maintaining old code you don't really want to keep maintaining, and ensuring the software works for the largest number of people. It is really easy to keep old code if it's just a big chunk of trivially #ifdefed logic that you can gate behind a configure option. Projects should feel encouraged to do precisely that whenever they feel a bit nervous about a change, it is very comforting to know that you can just tell users "no big deal, configure this brand new option off for now and we'll make it work in the next release". Your software still works, it's just the optional new thing that doesn't work. ;)