diff --git a/moodycamel/.gitignore b/moodycamel/.gitignore deleted file mode 100644 index 6aa8f8870..000000000 --- a/moodycamel/.gitignore +++ /dev/null @@ -1,26 +0,0 @@ -*.ipch -*.suo -*.user -*.sdf -*.opensdf -*.exe -*.pdb -*.vs -*.VC.db -build/bin/ -build/*.log -build/msvc14/*.log -build/msvc14/obj/ -build/msvc12/*.log -build/msvc12/obj/ -build/msvc11/*.log -build/msvc11/obj/ -build/xcode/build/ -tests/fuzztests/fuzztests.log -benchmarks/benchmarks.log -tests/CDSChecker/*.o -tests/CDSChecker/*.log -tests/CDSChecker/model-checker/ -tests/relacy/freelist.exe -tests/relacy/spmchash.exe -tests/relacy/log.txt diff --git a/moodycamel/LICENSE.md b/moodycamel/LICENSE.md old mode 100644 new mode 100755 index c4e7588a8..7b667d9e6 --- a/moodycamel/LICENSE.md +++ b/moodycamel/LICENSE.md @@ -1,15 +1,11 @@ -This license file applies to everything in this repository except that which -is explicitly annotated as being written by other authors, i.e. the Boost -queue (included in the benchmarks for comparison), Intel's TBB library (ditto), -the CDSChecker tool (used for verification), the Relacy model checker (ditto), -and Jeff Preshing's semaphore implementation (used in the blocking queue) which -has a zlib license (embedded in blockingconcurrentqueue.h). - ---- +This license applies to all the code in this repository except that written by third +parties, namely the files in benchmarks/ext, which have their own licenses, and Jeff +Preshing's semaphore implementation (used in the blocking queues) which has a zlib +license (embedded in atomicops.h). Simplified BSD License: -Copyright (c) 2013-2016, Cameron Desrochers. +Copyright (c) 2013-2021, Cameron Desrochers All rights reserved. Redistribution and use in source and binary forms, with or without modification, @@ -30,32 +26,3 @@ OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTE HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - ---- - -I have also chosen to dual-license under the Boost Software License as an alternative to -the Simplified BSD license above: - -Boost Software License - Version 1.0 - August 17th, 2003 - -Permission is hereby granted, free of charge, to any person or organization -obtaining a copy of the software and accompanying documentation covered by -this license (the "Software") to use, reproduce, display, distribute, -execute, and transmit the Software, and to prepare derivative works of the -Software, and to permit third-parties to whom the Software is furnished to -do so, all subject to the following: - -The copyright notices in the Software and this entire statement, including -the above license grant, this restriction and the following disclaimer, -must be included in all copies of the Software, in whole or in part, and -all derivative works of the Software, unless such copies or derivative -works are solely in the form of machine-executable object code generated by -a source language processor. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT -SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE -FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, -ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -DEALINGS IN THE SOFTWARE. diff --git a/moodycamel/README.md b/moodycamel/README.md old mode 100644 new mode 100755 index 327eb9ef6..ba87ecfae --- a/moodycamel/README.md +++ b/moodycamel/README.md @@ -1,486 +1,155 @@ -# moodycamel::ConcurrentQueue +# A single-producer, single-consumer lock-free queue for C++ -An industrial-strength lock-free queue for C++. +This mini-repository has my very own implementation of a lock-free queue (that I designed from scratch) for C++. -Note: If all you need is a single-producer, single-consumer queue, I have [one of those too][spsc]. +It only supports a two-thread use case (one consuming, and one producing). The threads can't switch roles, though +you could use this queue completely from a single thread if you wish (but that would sort of defeat the purpose!). -## Features +Note: If you need a general-purpose multi-producer, multi-consumer lock free queue, I have [one of those too][mpmc]. -- Knock-your-socks-off [blazing fast performance][benchmarks]. -- Single-header implementation. Just drop it in your project. -- Fully thread-safe lock-free queue. Use concurrently from any number of threads. -- C++11 implementation -- elements are moved (instead of copied) where possible. -- Templated, obviating the need to deal exclusively with pointers -- memory is managed for you. -- No artificial limitations on element types or maximum count. -- Memory can be allocated once up-front, or dynamically as needed. -- Fully portable (no assembly; all is done through standard C++11 primitives). -- Supports super-fast bulk operations. -- Includes a low-overhead blocking version (BlockingConcurrentQueue). -- Exception safe. - -## Reasons to use - -There are not that many full-fledged lock-free queues for C++. Boost has one, but it's limited to objects with trivial -assignment operators and trivial destructors, for example. -Intel's TBB queue isn't lock-free, and requires trivial constructors too. -There're many academic papers that implement lock-free queues in C++, but usable source code is -hard to find, and tests even more so. - -This queue not only has less limitations than others (for the most part), but [it's also faster][benchmarks]. -It's been fairly well-tested, and offers advanced features like **bulk enqueueing/dequeueing** -(which, with my new design, is much faster than one element at a time, approaching and even surpassing -the speed of a non-concurrent queue even under heavy contention). - -In short, there was a lock-free queue shaped hole in the C++ open-source universe, and I set out -to fill it with the fastest, most complete, and well-tested design and implementation I could. -The result is `moodycamel::ConcurrentQueue` :-) - -## Reasons *not* to use - -The fastest synchronization of all is the kind that never takes place. Fundamentally, -concurrent data structures require some synchronization, and that takes time. Every effort -was made, of course, to minimize the overhead, but if you can avoid sharing data between -threads, do so! - -Why use concurrent data structures at all, then? Because they're gosh darn convenient! (And, indeed, -sometimes sharing data concurrently is unavoidable.) - -My queue is **not linearizable** (see the next section on high-level design). The foundations of -its design assume that producers are independent; if this is not the case, and your producers -co-ordinate amongst themselves in some fashion, be aware that the elements won't necessarily -come out of the queue in the same order they were put in *relative to the ordering formed by that co-ordination* -(but they will still come out in the order they were put in by any *individual* producer). If this affects -your use case, you may be better off with another implementation; either way, it's an important limitation -to be aware of. - -My queue is also **not NUMA aware**, and does a lot of memory re-use internally, meaning it probably doesn't -scale particularly well on NUMA architectures; however, I don't know of any other lock-free queue that *is* -NUMA aware (except for [SALSA][salsa], which is very cool, but has no publicly available implementation that I know of). - -Finally, the queue is **not sequentially consistent**; there *is* a happens-before relationship between when an element is put -in the queue and when it comes out, but other things (such as pumping the queue until it's empty) require more thought -to get right in all eventualities, because explicit memory ordering may have to be done to get the desired effect. In other words, -it can sometimes be difficult to use the queue correctly. This is why it's a good idea to follow the [samples][samples.md] where possible. -On the other hand, the upside of this lack of sequential consistency is better performance. - -## High-level design - -Elements are stored internally using contiguous blocks instead of linked lists for better performance. -The queue is made up of a collection of sub-queues, one for each producer. When a consumer -wants to dequeue an element, it checks all the sub-queues until it finds one that's not empty. -All of this is largely transparent to the user of the queue, however -- it mostly just worksTM. - -One particular consequence of this design, however, (which seems to be non-intuitive) is that if two producers -enqueue at the same time, there is no defined ordering between the elements when they're later dequeued. -Normally this is fine, because even with a fully linearizable queue there'd be a race between the producer -threads and so you couldn't rely on the ordering anyway. However, if for some reason you do extra explicit synchronization -between the two producer threads yourself, thus defining a total order between enqueue operations, you might expect -that the elements would come out in the same total order, which is a guarantee my queue does not offer. At that -point, though, there semantically aren't really two separate producers, but rather one that happens to be spread -across multiple threads. In this case, you can still establish a total ordering with my queue by creating -a single producer token, and using that from both threads to enqueue (taking care to synchronize access to the token, -of course, but there was already extra synchronization involved anyway). - -I've written a more detailed [overview of the internal design][blog], as well as [the full -nitty-gritty details of the design][design], on my blog. Finally, the -[source][source] itself is available for perusal for those interested in its implementation. - -## Basic use - -The entire queue's implementation is contained in **one header**, [`concurrentqueue.h`][concurrentqueue.h]. -Simply download and include that to use the queue. The blocking version is in a separate header, -[`blockingconcurrentqueue.h`][blockingconcurrentqueue.h], that depends on the first. -The implementation makes use of certain key C++11 features, so it requires a fairly recent compiler -(e.g. VS2012+ or g++ 4.8; note that g++ 4.6 has a known bug with `std::atomic` and is thus not supported). -The algorithm implementations themselves are platform independent. - -Use it like you would any other templated queue, with the exception that you can use -it from many threads at once :-) - -Simple example: - - #include "concurrentqueue.h" - - moodycamel::ConcurrentQueue q; - q.enqueue(25); - - int item; - bool found = q.try_dequeue(item); - assert(found && item == 25); - -Description of basic methods: -- `ConcurrentQueue(size_t initialSizeEstimate)` - Constructor which optionally accepts an estimate of the number of elements the queue will hold -- `enqueue(T&& item)` - Enqueues one item, allocating extra space if necessary -- `try_enqueue(T&& item)` - Enqueues one item, but only if enough memory is already allocated -- `try_dequeue(T& item)` - Dequeues one item, returning true if an item was found or false if the queue appeared empty - -Note that it is up to the user to ensure that the queue object is completely constructed before -being used by any other threads (this includes making the memory effects of construction -visible, possibly via a memory barrier). Similarly, it's important that all threads have -finished using the queue (and the memory effects have fully propagated) before it is -destructed. - -There's usually two versions of each method, one "explicit" version that takes a user-allocated per-producer or -per-consumer token, and one "implicit" version that works without tokens. Using the explicit methods is almost -always faster (though not necessarily by a huge factor). Apart from performance, the primary distinction between them -is their sub-queue allocation behaviour for enqueue operations: Using the implicit enqueue methods causes an -automatically-allocated thread-local producer sub-queue to be allocated (it is marked for reuse once the thread exits). -Explicit producers, on the other hand, are tied directly to their tokens' lifetimes (and are also recycled as needed). - -Full API (pseudocode): - - # Allocates more memory if necessary - enqueue(item) : bool - enqueue(prod_token, item) : bool - enqueue_bulk(item_first, count) : bool - enqueue_bulk(prod_token, item_first, count) : bool - - # Fails if not enough memory to enqueue - try_enqueue(item) : bool - try_enqueue(prod_token, item) : bool - try_enqueue_bulk(item_first, count) : bool - try_enqueue_bulk(prod_token, item_first, count) : bool - - # Attempts to dequeue from the queue (never allocates) - try_dequeue(item&) : bool - try_dequeue(cons_token, item&) : bool - try_dequeue_bulk(item_first, max) : size_t - try_dequeue_bulk(cons_token, item_first, max) : size_t - - # If you happen to know which producer you want to dequeue from - try_dequeue_from_producer(prod_token, item&) : bool - try_dequeue_bulk_from_producer(prod_token, item_first, max) : size_t - - # A not-necessarily-accurate count of the total number of elements - size_approx() : size_t - -## Blocking version - -As mentioned above, a full blocking wrapper of the queue is provided that adds -`wait_dequeue` and `wait_dequeue_bulk` methods in addition to the regular interface. -This wrapper is extremely low-overhead, but slightly less fast than the non-blocking -queue (due to the necessary bookkeeping involving a lightweight semaphore). - -There are also timed versions that allow a timeout to be specified (either in microseconds -or with a `std::chrono` object). - -The only major caveat with the blocking version is that you must be careful not to -destroy the queue while somebody is waiting on it. This generally means you need to -know for certain that another element is going to come along before you call one of -the blocking methods. (To be fair, the non-blocking version cannot be destroyed while -in use either, but it can be easier to coordinate the cleanup.) - -Blocking example: - - #include "blockingconcurrentqueue.h" - - moodycamel::BlockingConcurrentQueue q; - std::thread producer([&]() { - for (int i = 0; i != 100; ++i) { - std::this_thread::sleep_for(std::chrono::milliseconds(i % 10)); - q.enqueue(i); - } - }); - std::thread consumer([&]() { - for (int i = 0; i != 100; ++i) { - int item; - q.wait_dequeue(item); - assert(item == i); - - if (q.wait_dequeue_timed(item, std::chrono::milliseconds(5))) { - ++i; - assert(item == i); - } - } - }); - producer.join(); - consumer.join(); - - assert(q.size_approx() == 0); +This repository also includes a [circular-buffer SPSC queue][circular] which supports blocking on enqueue as well as dequeue. -## Advanced features -#### Tokens +## Features -The queue can take advantage of extra per-producer and per-consumer storage if -it's available to speed up its operations. This takes the form of "tokens": -You can create a consumer token and/or a producer token for each thread or task -(tokens themselves are not thread-safe), and use the methods that accept a token -as their first parameter: +- [Blazing fast][benchmarks] +- Compatible with C++11 (supports moving objects instead of making copies) +- Fully generic (templated container of any type) -- just like `std::queue`, you never need to allocate memory for elements yourself + (which saves you the hassle of writing a lock-free memory manager to hold the elements you're queueing) +- Allocates memory up front, in contiguous blocks +- Provides a `try_enqueue` method which is guaranteed never to allocate memory (the queue starts with an initial capacity) +- Also provides an `enqueue` method which can dynamically grow the size of the queue as needed +- Also provides `try_emplace`/`emplace` convenience methods +- Has a blocking version with `wait_dequeue` +- Completely "wait-free" (no compare-and-swap loop). Enqueue and dequeue are always O(1) (not counting memory allocation) +- On x86, the memory barriers compile down to no-ops, meaning enqueue and dequeue are just a simple series of loads and stores (and branches) - moodycamel::ConcurrentQueue q; - - moodycamel::ProducerToken ptok(q); - q.enqueue(ptok, 17); - - moodycamel::ConsumerToken ctok(q); - int item; - q.try_dequeue(ctok, item); - assert(item == 17); -If you happen to know which producer you want to consume from (e.g. in -a single-producer, multi-consumer scenario), you can use the `try_dequeue_from_producer` -methods, which accept a producer token instead of a consumer token, and cut some overhead. +## Use -Note that tokens work with the blocking version of the queue too. +Simply drop the readerwriterqueue.h (or readerwritercircularbuffer.h) and atomicops.h files into your source code and include them :-) +A modern compiler is required (MSVC2010+, GCC 4.7+, ICC 13+, or any C++11 compliant compiler should work). -When producing or consuming many elements, the most efficient way is to: +Note: If you're using GCC, you really do need GCC 4.7 or above -- [4.6 has a bug][gcc46bug] that prevents the atomic fence primitives +from working correctly. -1. Use the bulk methods of the queue with tokens -2. Failing that, use the bulk methods without tokens -3. Failing that, use the single-item methods with tokens -4. Failing that, use the single-item methods without tokens +Example: -Having said that, don't create tokens willy-nilly -- ideally there would be -one token (of each kind) per thread. The queue will work with what it is -given, but it performs best when used with tokens. +```cpp +using namespace moodycamel; -Note that tokens aren't actually tied to any given thread; it's not technically -required that they be local to the thread, only that they be used by a single -producer/consumer at a time. +ReaderWriterQueue q(100); // Reserve space for at least 100 elements up front -#### Bulk operations +q.enqueue(17); // Will allocate memory if the queue is full +bool succeeded = q.try_enqueue(18); // Will only succeed if the queue has an empty slot (never allocates) +assert(succeeded); -Thanks to the [novel design][blog] of the queue, it's just as easy to enqueue/dequeue multiple -items as it is to do one at a time. This means that overhead can be cut drastically for -bulk operations. Example syntax: +int number; +succeeded = q.try_dequeue(number); // Returns false if the queue was empty - moodycamel::ConcurrentQueue q; +assert(succeeded && number == 17); - int items[] = { 1, 2, 3, 4, 5 }; - q.enqueue_bulk(items, 5); - - int results[5]; // Could also be any iterator - size_t count = q.try_dequeue_bulk(results, 5); - for (size_t i = 0; i != count; ++i) { - assert(results[i] == items[i]); - } +// You can also peek at the front item of the queue (consumer only) +int* front = q.peek(); +assert(*front == 18); +succeeded = q.try_dequeue(number); +assert(succeeded && number == 18); +front = q.peek(); +assert(front == nullptr); // Returns nullptr if the queue was empty +``` -#### Preallocation (correctly using `try_enqueue`) +The blocking version has the exact same API, with the addition of `wait_dequeue` and +`wait_dequeue_timed` methods: -`try_enqueue`, unlike just plain `enqueue`, will never allocate memory. If there's not enough room in the -queue, it simply returns false. The key to using this method properly, then, is to ensure enough space is -pre-allocated for your desired maximum element count. +```cpp +BlockingReaderWriterQueue q; -The constructor accepts a count of the number of elements that it should reserve space for. Because the -queue works with blocks of elements, however, and not individual elements themselves, the value to pass -in order to obtain an effective number of pre-allocated element slots is non-obvious. +std::thread reader([&]() { + int item; +#if 1 + for (int i = 0; i != 100; ++i) { + // Fully-blocking: + q.wait_dequeue(item); + } +#else + for (int i = 0; i != 100; ) { + // Blocking with timeout + if (q.wait_dequeue_timed(item, std::chrono::milliseconds(5))) + ++i; + } +#endif +}); +std::thread writer([&]() { + for (int i = 0; i != 100; ++i) { + q.enqueue(i); + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +}); +writer.join(); +reader.join(); -First, be aware that the count passed is rounded up to the next multiple of the block size. Note that the -default block size is 32 (this can be changed via the traits). Second, once a slot in a block has been -enqueued to, that slot cannot be re-used until the rest of the block has completely been completely filled -up and then completely emptied. This affects the number of blocks you need in order to account for the -overhead of partially-filled blocks. Third, each producer (whether implicit or explicit) claims and recycles -blocks in a different manner, which again affects the number of blocks you need to account for a desired number of -usable slots. +assert(q.size_approx() == 0); +``` + +Note that `wait_dequeue` will block indefinitely while the queue is empty; this +means care must be taken to only call `wait_dequeue` if you're sure another element +will come along eventually, or if the queue has a static lifetime. This is because +destroying the queue while a thread is waiting on it will invoke undefined behaviour. -Suppose you want the queue to be able to hold at least `N` elements at any given time. Without delving too -deep into the rather arcane implementation details, here are some simple formulas for the number of elements -to request for pre-allocation in such a case. Note the division is intended to be arithmetic division and not -integer division (in order for `ceil()` to work). +The blocking circular buffer has a fixed number of slots, but is otherwise quite similar to +use: -For explicit producers (using tokens to enqueue): +```cpp +BlockingReaderWriterCircularBuffer q(1024); // pass initial capacity - (ceil(N / BLOCK_SIZE) + 1) * MAX_NUM_PRODUCERS * BLOCK_SIZE +q.try_enqueue(1); +int number; +q.try_dequeue(number); +assert(number == 1); -For implicit producers (no tokens): +q.wait_enqueue(123); +q.wait_dequeue(number); +assert(number == 123); - (ceil(N / BLOCK_SIZE) - 1 + 2 * MAX_NUM_PRODUCERS) * BLOCK_SIZE +q.wait_dequeue_timed(number, std::chrono::milliseconds(10)); +``` -When using mixed producer types: - ((ceil(N / BLOCK_SIZE) - 1) * (MAX_EXPLICIT_PRODUCERS + 1) + 2 * (MAX_IMPLICIT_PRODUCERS + MAX_EXPLICIT_PRODUCERS)) * BLOCK_SIZE +## CMake installation +As an alternative to including the source files in your project directly, +you can use CMake to install the library in your system's include directory: -If these formulas seem rather inconvenient, you can use the constructor overload that accepts the minimum -number of elements (`N`) and the maximum number of explicit and implicit producers directly, and let it do the -computation for you. +``` +mkdir build +cd build +cmake .. +make install +``` -Finally, it's important to note that because the queue is only eventually consistent and takes advantage of -weak memory ordering for speed, there's always a possibility that under contention `try_enqueue` will fail -even if the queue is correctly pre-sized for the desired number of elements. (e.g. A given thread may think that -the queue's full even when that's no longer the case.) So no matter what, you still need to handle the failure -case (perhaps looping until it succeeds), unless you don't mind dropping elements. +Then, you can include it from your source code: +``` +#include +``` -#### Exception safety +## Disclaimers -The queue is exception safe, and will never become corrupted if used with a type that may throw exceptions. -The queue itself never throws any exceptions (operations fail gracefully (return false) if memory allocation -fails instead of throwing `std::bad_alloc`). +The queue should only be used on platforms where aligned integer and pointer access is atomic; fortunately, that +includes all modern processors (e.g. x86/x86-64, ARM, and PowerPC). *Not* for use with a DEC Alpha processor (which has very weak memory ordering) :-) -It is important to note that the guarantees of exception safety only hold if the element type never throws -from its destructor, and that any iterators passed into the queue (for bulk operations) never throw either. -Note that in particular this means `std::back_inserter` iterators must be used with care, since the vector -being inserted into may need to allocate and throw a `std::bad_alloc` exception from inside the iterator; -so be sure to reserve enough capacity in the target container first if you do this. +Note that it's only been tested on x86(-64); if someone has access to other processors I'd love to run some tests on +anything that's not x86-based. -The guarantees are presently as follows: -- Enqueue operations are rolled back completely if an exception is thrown from an element's constructor. - For bulk enqueue operations, this means that elements are copied instead of moved (in order to avoid - having only some of the objects be moved in the event of an exception). Non-bulk enqueues always use - the move constructor if one is available. -- If the assignment operator throws during a dequeue operation (both single and bulk), the element(s) are - considered dequeued regardless. In such a case, the dequeued elements are all properly destructed before - the exception is propagated, but there's no way to get the elements themselves back. -- Any exception that is thrown is propagated up the call stack, at which point the queue is in a consistent - state. +## More info -Note: If any of your type's copy constructors/move constructors/assignment operators don't throw, be sure -to annotate them with `noexcept`; this will avoid the exception-checking overhead in the queue where possible -(even with zero-cost exceptions, there's still a code size impact that has to be taken into account). +See the [LICENSE.md][license] file for the license (simplified BSD). -#### Traits +My [blog post][blog] introduces the context that led to this code, and may be of interest if you're curious +about lock-free programming. -The queue also supports a traits template argument which defines various types, constants, -and the memory allocation and deallocation functions that are to be used by the queue. The typical pattern -to providing your own traits is to create a class that inherits from the default traits -and override only the values you wish to change. Example: - struct MyTraits : public moodycamel::ConcurrentQueueDefaultTraits - { - static const size_t BLOCK_SIZE = 256; // Use bigger blocks - }; - - moodycamel::ConcurrentQueue q; - -#### How to dequeue types without calling the constructor - -The normal way to dequeue an item is to pass in an existing object by reference, which -is then assigned to internally by the queue (using the move-assignment operator if possible). -This can pose a problem for types that are -expensive to construct or don't have a default constructor; fortunately, there is a simple -workaround: Create a wrapper class that copies the memory contents of the object when it -is assigned by the queue (a poor man's move, essentially). Note that this only works if -the object contains no internal pointers. Example: - - struct MyObjectMover { - inline void operator=(MyObject&& obj) - { - std::memcpy(data, &obj, sizeof(MyObject)); - - // TODO: Cleanup obj so that when it's destructed by the queue - // it doesn't corrupt the data of the object we just moved it into - } - - inline MyObject& obj() { return *reinterpret_cast(data); } - - private: - align(alignof(MyObject)) char data[sizeof(MyObject)]; - }; - -A less dodgy alternative, if moves are cheap but default construction is not, is to use a -wrapper that defers construction until the object is assigned, enabling use of the move -constructor: - - struct MyObjectMover { - inline void operator=(MyObject&& x) { - new (data) MyObject(std::move(x)); - created = true; - } - - inline MyObject& obj() { - assert(created); - return *reinterpret_cast(data); - } - - ~MyObjectMover() { - if (created) - obj().~MyObject(); - } - - private: - align(alignof(MyObject)) char data[sizeof(MyObject)]; - bool created = false; - }; - - -## Samples - -There are some more detailed samples [here][samples.md]. The source of -the [unit tests][unittest-src] and [benchmarks][benchmark-src] are available for reference as well. - -## Benchmarks - -See my blog post for some [benchmark results][benchmarks] (including versus `boost::lockfree::queue` and `tbb::concurrent_queue`), -or run the benchmarks yourself (requires MinGW and certain GnuWin32 utilities to build on Windows, or a recent -g++ on Linux): - - cd build - make benchmarks - bin/benchmarks - -The short version of the benchmarks is that it's so fast (especially the bulk methods), that if you're actually -using the queue to *do* anything, the queue won't be your bottleneck. - -## Tests (and bugs) - -I've written quite a few unit tests as well as a randomized long-running fuzz tester. I also ran the -core queue algorithm through the [CDSChecker][cdschecker] C++11 memory model model checker. Some of the -inner algorithms were tested separately using the [Relacy][relacy] model checker, and full integration -tests were also performed with Relacy. -I've tested -on Linux (Fedora 19) and Windows (7), but only on x86 processors so far (Intel and AMD). The code was -written to be platform-independent, however, and should work across all processors and OSes. - -Due to the complexity of the implementation and the difficult-to-test nature of lock-free code in general, -there may still be bugs. If anyone is seeing buggy behaviour, I'd like to hear about it! (Especially if -a unit test for it can be cooked up.) Just open an issue on GitHub. - -## License - -I'm releasing the source of this repository (with the exception of third-party code, i.e. the Boost queue -(used in the benchmarks for comparison), Intel's TBB library (ditto), CDSChecker, Relacy, and Jeff Preshing's -cross-platform semaphore, which all have their own licenses) -under a simplified BSD license. I'm also dual-licensing under the Boost Software License. -See the [LICENSE.md][license] file for more details. - -Note that lock-free programming is a patent minefield, and this code may very -well violate a pending patent (I haven't looked), though it does not to my present knowledge. -I did design and implement this queue from scratch. - -## Diving into the code - -If you're interested in the source code itself, it helps to have a rough idea of how it's laid out. This -section attempts to describe that. - -The queue is formed of several basic parts (listed here in roughly the order they appear in the source). There's the -helper functions (e.g. for rounding to a power of 2). There's the default traits of the queue, which contain the -constants and malloc/free functions used by the queue. There's the producer and consumer tokens. Then there's the queue's -public API itself, starting with the constructor, destructor, and swap/assignment methods. There's the public enqueue methods, -which are all wrappers around a small set of private enqueue methods found later on. There's the dequeue methods, which are -defined inline and are relatively straightforward. - -Then there's all the main internal data structures. First, there's a lock-free free list, used for recycling spent blocks (elements -are enqueued to blocks internally). Then there's the block structure itself, which has two different ways of tracking whether -it's fully emptied or not (remember, given two parallel consumers, there's no way to know which one will finish first) depending on where it's used. -Then there's a small base class for the two types of internal SPMC producer queues (one for explicit producers that holds onto memory -but attempts to be faster, and one for implicit ones which attempt to recycle more memory back into the parent but is a little slower). -The explicit producer is defined first, then the implicit one. They both contain the same general four methods: One to enqueue, one to -dequeue, one to enqueue in bulk, and one to dequeue in bulk. (Obviously they have constructors and destructors too, and helper methods.) -The main difference between them is how the block handling is done (they both use the same blocks, but in different ways, and map indices -to them in different ways). - -Finally, there's the miscellaneous internal methods: There's the ones that handle the initial block pool (populated when the queue is constructed), -and an abstract block pool that comprises the initial pool and any blocks on the free list. There's ones that handle the producer list -(a lock-free add-only linked list of all the producers in the system). There's ones that handle the implicit producer lookup table (which -is really a sort of specialized TLS lookup). And then there's some helper methods for allocating and freeing objects, and the data members -of the queue itself, followed lastly by the free-standing swap functions. - - -[blog]: http://moodycamel.com/blog/2014/a-fast-general-purpose-lock-free-queue-for-c++ -[design]: http://moodycamel.com/blog/2014/detailed-design-of-a-lock-free-queue -[samples.md]: https://github.com/cameron314/concurrentqueue/blob/master/samples.md -[source]: https://github.com/cameron314/concurrentqueue -[concurrentqueue.h]: https://github.com/cameron314/concurrentqueue/blob/master/concurrentqueue.h -[blockingconcurrentqueue.h]: https://github.com/cameron314/concurrentqueue/blob/master/blockingconcurrentqueue.h -[unittest-src]: https://github.com/cameron314/concurrentqueue/tree/master/tests/unittests -[benchmarks]: http://moodycamel.com/blog/2014/a-fast-general-purpose-lock-free-queue-for-c++#benchmarks -[benchmark-src]: https://github.com/cameron314/concurrentqueue/tree/master/benchmarks -[license]: https://github.com/cameron314/concurrentqueue/blob/master/LICENSE.md -[cdschecker]: http://demsky.eecs.uci.edu/c11modelchecker.html -[relacy]: http://www.1024cores.net/home/relacy-race-detector -[spsc]: https://github.com/cameron314/readerwriterqueue -[salsa]: http://webee.technion.ac.il/~idish/ftp/spaa049-gidron.pdf +[blog]: http://moodycamel.com/blog/2013/a-fast-lock-free-queue-for-c++ +[license]: LICENSE.md +[benchmarks]: http://moodycamel.com/blog/2013/a-fast-lock-free-queue-for-c++#benchmarks +[gcc46bug]: http://stackoverflow.com/questions/16429669/stdatomic-thread-fence-has-undefined-reference +[mpmc]: https://github.com/cameron314/concurrentqueue +[circular]: readerwritercircularbuffer.h diff --git a/moodycamel/atomicops.h b/moodycamel/atomicops.h new file mode 100755 index 000000000..2bb37963d --- /dev/null +++ b/moodycamel/atomicops.h @@ -0,0 +1,690 @@ +// ©2013-2016 Cameron Desrochers. +// Distributed under the simplified BSD license (see the license file that +// should have come with this header). +// Uses Jeff Preshing's semaphore implementation (under the terms of its +// separate zlib license, embedded below). + +#pragma once + +// Provides portable (VC++2010+, Intel ICC 13, GCC 4.7+, and anything C++11 compliant) implementation +// of low-level memory barriers, plus a few semi-portable utility macros (for inlining and alignment). +// Also has a basic atomic type (limited to hardware-supported atomics with no memory ordering guarantees). +// Uses the AE_* prefix for macros (historical reasons), and the "moodycamel" namespace for symbols. + +#include +#include +#include +#include +#include +#include + +// Platform detection +#if defined(__INTEL_COMPILER) +#define AE_ICC +#elif defined(_MSC_VER) +#define AE_VCPP +#elif defined(__GNUC__) +#define AE_GCC +#endif + +#if defined(_M_IA64) || defined(__ia64__) +#define AE_ARCH_IA64 +#elif defined(_WIN64) || defined(__amd64__) || defined(_M_X64) || defined(__x86_64__) +#define AE_ARCH_X64 +#elif defined(_M_IX86) || defined(__i386__) +#define AE_ARCH_X86 +#elif defined(_M_PPC) || defined(__powerpc__) +#define AE_ARCH_PPC +#else +#define AE_ARCH_UNKNOWN +#endif + + +// AE_UNUSED +#define AE_UNUSED(x) ((void)x) + +// AE_NO_TSAN/AE_TSAN_ANNOTATE_* +#if defined(__has_feature) +#if __has_feature(thread_sanitizer) +#if __cplusplus >= 201703L // inline variables require C++17 +namespace moodycamel { inline int ae_tsan_global; } +#define AE_TSAN_ANNOTATE_RELEASE() AnnotateHappensBefore(__FILE__, __LINE__, (void *)(&::moodycamel::ae_tsan_global)) +#define AE_TSAN_ANNOTATE_ACQUIRE() AnnotateHappensAfter(__FILE__, __LINE__, (void *)(&::moodycamel::ae_tsan_global)) +extern "C" void AnnotateHappensBefore(const char*, int, void*); +extern "C" void AnnotateHappensAfter(const char*, int, void*); +#else // when we can't work with tsan, attempt to disable its warnings +#define AE_NO_TSAN __attribute__((no_sanitize("thread"))) +#endif +#endif +#endif +#ifndef AE_NO_TSAN +#define AE_NO_TSAN +#endif +#ifndef AE_TSAN_ANNOTATE_RELEASE +#define AE_TSAN_ANNOTATE_RELEASE() +#define AE_TSAN_ANNOTATE_ACQUIRE() +#endif + + +// AE_FORCEINLINE +#if defined(AE_VCPP) || defined(AE_ICC) +#define AE_FORCEINLINE __forceinline +#elif defined(AE_GCC) +//#define AE_FORCEINLINE __attribute__((always_inline)) +#define AE_FORCEINLINE inline +#else +#define AE_FORCEINLINE inline +#endif + + +// AE_ALIGN +#if defined(AE_VCPP) || defined(AE_ICC) +#define AE_ALIGN(x) __declspec(align(x)) +#elif defined(AE_GCC) +#define AE_ALIGN(x) __attribute__((aligned(x))) +#else +// Assume GCC compliant syntax... +#define AE_ALIGN(x) __attribute__((aligned(x))) +#endif + + +// Portable atomic fences implemented below: + +namespace moodycamel { + +enum memory_order { + memory_order_relaxed, + memory_order_acquire, + memory_order_release, + memory_order_acq_rel, + memory_order_seq_cst, + + // memory_order_sync: Forces a full sync: + // #LoadLoad, #LoadStore, #StoreStore, and most significantly, #StoreLoad + memory_order_sync = memory_order_seq_cst +}; + +} // end namespace moodycamel + +#if (defined(AE_VCPP) && (_MSC_VER < 1700 || defined(__cplusplus_cli))) || (defined(AE_ICC) && __INTEL_COMPILER < 1600) +// VS2010 and ICC13 don't support std::atomic_*_fence, implement our own fences + +#include + +#if defined(AE_ARCH_X64) || defined(AE_ARCH_X86) +#define AeFullSync _mm_mfence +#define AeLiteSync _mm_mfence +#elif defined(AE_ARCH_IA64) +#define AeFullSync __mf +#define AeLiteSync __mf +#elif defined(AE_ARCH_PPC) +#include +#define AeFullSync __sync +#define AeLiteSync __lwsync +#endif + + +#ifdef AE_VCPP +#pragma warning(push) +#pragma warning(disable: 4365) // Disable erroneous 'conversion from long to unsigned int, signed/unsigned mismatch' error when using `assert` +#ifdef __cplusplus_cli +#pragma managed(push, off) +#endif +#endif + +namespace moodycamel { + +AE_FORCEINLINE void compiler_fence(memory_order order) AE_NO_TSAN +{ + switch (order) { + case memory_order_relaxed: break; + case memory_order_acquire: _ReadBarrier(); break; + case memory_order_release: _WriteBarrier(); break; + case memory_order_acq_rel: _ReadWriteBarrier(); break; + case memory_order_seq_cst: _ReadWriteBarrier(); break; + default: assert(false); + } +} + +// x86/x64 have a strong memory model -- all loads and stores have +// acquire and release semantics automatically (so only need compiler +// barriers for those). +#if defined(AE_ARCH_X86) || defined(AE_ARCH_X64) +AE_FORCEINLINE void fence(memory_order order) AE_NO_TSAN +{ + switch (order) { + case memory_order_relaxed: break; + case memory_order_acquire: _ReadBarrier(); break; + case memory_order_release: _WriteBarrier(); break; + case memory_order_acq_rel: _ReadWriteBarrier(); break; + case memory_order_seq_cst: + _ReadWriteBarrier(); + AeFullSync(); + _ReadWriteBarrier(); + break; + default: assert(false); + } +} +#else +AE_FORCEINLINE void fence(memory_order order) AE_NO_TSAN +{ + // Non-specialized arch, use heavier memory barriers everywhere just in case :-( + switch (order) { + case memory_order_relaxed: + break; + case memory_order_acquire: + _ReadBarrier(); + AeLiteSync(); + _ReadBarrier(); + break; + case memory_order_release: + _WriteBarrier(); + AeLiteSync(); + _WriteBarrier(); + break; + case memory_order_acq_rel: + _ReadWriteBarrier(); + AeLiteSync(); + _ReadWriteBarrier(); + break; + case memory_order_seq_cst: + _ReadWriteBarrier(); + AeFullSync(); + _ReadWriteBarrier(); + break; + default: assert(false); + } +} +#endif +} // end namespace moodycamel +#else +// Use standard library of atomics +#include + +namespace moodycamel { + +AE_FORCEINLINE void compiler_fence(memory_order order) AE_NO_TSAN +{ + switch (order) { + case memory_order_relaxed: break; + case memory_order_acquire: std::atomic_signal_fence(std::memory_order_acquire); break; + case memory_order_release: std::atomic_signal_fence(std::memory_order_release); break; + case memory_order_acq_rel: std::atomic_signal_fence(std::memory_order_acq_rel); break; + case memory_order_seq_cst: std::atomic_signal_fence(std::memory_order_seq_cst); break; + default: assert(false); + } +} + +AE_FORCEINLINE void fence(memory_order order) AE_NO_TSAN +{ + switch (order) { + case memory_order_relaxed: break; + case memory_order_acquire: AE_TSAN_ANNOTATE_ACQUIRE(); std::atomic_thread_fence(std::memory_order_acquire); break; + case memory_order_release: AE_TSAN_ANNOTATE_RELEASE(); std::atomic_thread_fence(std::memory_order_release); break; + case memory_order_acq_rel: AE_TSAN_ANNOTATE_ACQUIRE(); AE_TSAN_ANNOTATE_RELEASE(); std::atomic_thread_fence(std::memory_order_acq_rel); break; + case memory_order_seq_cst: AE_TSAN_ANNOTATE_ACQUIRE(); AE_TSAN_ANNOTATE_RELEASE(); std::atomic_thread_fence(std::memory_order_seq_cst); break; + default: assert(false); + } +} + +} // end namespace moodycamel + +#endif + + +#if !defined(AE_VCPP) || (_MSC_VER >= 1700 && !defined(__cplusplus_cli)) +#define AE_USE_STD_ATOMIC_FOR_WEAK_ATOMIC +#endif + +#ifdef AE_USE_STD_ATOMIC_FOR_WEAK_ATOMIC +#include +#endif +#include + +// WARNING: *NOT* A REPLACEMENT FOR std::atomic. READ CAREFULLY: +// Provides basic support for atomic variables -- no memory ordering guarantees are provided. +// The guarantee of atomicity is only made for types that already have atomic load and store guarantees +// at the hardware level -- on most platforms this generally means aligned pointers and integers (only). +namespace moodycamel { +template +class weak_atomic +{ +public: + AE_NO_TSAN weak_atomic() : value() { } +#ifdef AE_VCPP +#pragma warning(push) +#pragma warning(disable: 4100) // Get rid of (erroneous) 'unreferenced formal parameter' warning +#endif + template AE_NO_TSAN weak_atomic(U&& x) : value(std::forward(x)) { } +#ifdef __cplusplus_cli + // Work around bug with universal reference/nullptr combination that only appears when /clr is on + AE_NO_TSAN weak_atomic(nullptr_t) : value(nullptr) { } +#endif + AE_NO_TSAN weak_atomic(weak_atomic const& other) : value(other.load()) { } + AE_NO_TSAN weak_atomic(weak_atomic&& other) : value(std::move(other.load())) { } +#ifdef AE_VCPP +#pragma warning(pop) +#endif + + AE_FORCEINLINE operator T() const AE_NO_TSAN { return load(); } + + +#ifndef AE_USE_STD_ATOMIC_FOR_WEAK_ATOMIC + template AE_FORCEINLINE weak_atomic const& operator=(U&& x) AE_NO_TSAN { value = std::forward(x); return *this; } + AE_FORCEINLINE weak_atomic const& operator=(weak_atomic const& other) AE_NO_TSAN { value = other.value; return *this; } + + AE_FORCEINLINE T load() const AE_NO_TSAN { return value; } + + AE_FORCEINLINE T fetch_add_acquire(T increment) AE_NO_TSAN + { +#if defined(AE_ARCH_X64) || defined(AE_ARCH_X86) + if (sizeof(T) == 4) return _InterlockedExchangeAdd((long volatile*)&value, (long)increment); +#if defined(_M_AMD64) + else if (sizeof(T) == 8) return _InterlockedExchangeAdd64((long long volatile*)&value, (long long)increment); +#endif +#else +#error Unsupported platform +#endif + assert(false && "T must be either a 32 or 64 bit type"); + return value; + } + + AE_FORCEINLINE T fetch_add_release(T increment) AE_NO_TSAN + { +#if defined(AE_ARCH_X64) || defined(AE_ARCH_X86) + if (sizeof(T) == 4) return _InterlockedExchangeAdd((long volatile*)&value, (long)increment); +#if defined(_M_AMD64) + else if (sizeof(T) == 8) return _InterlockedExchangeAdd64((long long volatile*)&value, (long long)increment); +#endif +#else +#error Unsupported platform +#endif + assert(false && "T must be either a 32 or 64 bit type"); + return value; + } +#else + template + AE_FORCEINLINE weak_atomic const& operator=(U&& x) AE_NO_TSAN + { + value.store(std::forward(x), std::memory_order_relaxed); + return *this; + } + + AE_FORCEINLINE weak_atomic const& operator=(weak_atomic const& other) AE_NO_TSAN + { + value.store(other.value.load(std::memory_order_relaxed), std::memory_order_relaxed); + return *this; + } + + AE_FORCEINLINE T load() const AE_NO_TSAN { return value.load(std::memory_order_relaxed); } + + AE_FORCEINLINE T fetch_add_acquire(T increment) AE_NO_TSAN + { + return value.fetch_add(increment, std::memory_order_acquire); + } + + AE_FORCEINLINE T fetch_add_release(T increment) AE_NO_TSAN + { + return value.fetch_add(increment, std::memory_order_release); + } +#endif + + +private: +#ifndef AE_USE_STD_ATOMIC_FOR_WEAK_ATOMIC + // No std::atomic support, but still need to circumvent compiler optimizations. + // `volatile` will make memory access slow, but is guaranteed to be reliable. + volatile T value; +#else + std::atomic value; +#endif +}; + +} // end namespace moodycamel + + + +// Portable single-producer, single-consumer semaphore below: + +#if defined(_WIN32) +// Avoid including windows.h in a header; we only need a handful of +// items, so we'll redeclare them here (this is relatively safe since +// the API generally has to remain stable between Windows versions). +// I know this is an ugly hack but it still beats polluting the global +// namespace with thousands of generic names or adding a .cpp for nothing. +extern "C" { + struct _SECURITY_ATTRIBUTES; + __declspec(dllimport) void* __stdcall CreateSemaphoreW(_SECURITY_ATTRIBUTES* lpSemaphoreAttributes, long lInitialCount, long lMaximumCount, const wchar_t* lpName); + __declspec(dllimport) int __stdcall CloseHandle(void* hObject); + __declspec(dllimport) unsigned long __stdcall WaitForSingleObject(void* hHandle, unsigned long dwMilliseconds); + __declspec(dllimport) int __stdcall ReleaseSemaphore(void* hSemaphore, long lReleaseCount, long* lpPreviousCount); +} +#elif defined(__MACH__) +#include +#elif defined(__unix__) +#include +#endif + +namespace moodycamel +{ + // Code in the spsc_sema namespace below is an adaptation of Jeff Preshing's + // portable + lightweight semaphore implementations, originally from + // https://github.com/preshing/cpp11-on-multicore/blob/master/common/sema.h + // LICENSE: + // Copyright (c) 2015 Jeff Preshing + // + // This software is provided 'as-is', without any express or implied + // warranty. In no event will the authors be held liable for any damages + // arising from the use of this software. + // + // Permission is granted to anyone to use this software for any purpose, + // including commercial applications, and to alter it and redistribute it + // freely, subject to the following restrictions: + // + // 1. The origin of this software must not be misrepresented; you must not + // claim that you wrote the original software. If you use this software + // in a product, an acknowledgement in the product documentation would be + // appreciated but is not required. + // 2. Altered source versions must be plainly marked as such, and must not be + // misrepresented as being the original software. + // 3. This notice may not be removed or altered from any source distribution. + namespace spsc_sema + { +#if defined(_WIN32) + class Semaphore + { + private: + void* m_hSema; + + Semaphore(const Semaphore& other); + Semaphore& operator=(const Semaphore& other); + + public: + AE_NO_TSAN Semaphore(int initialCount = 0) : m_hSema() + { + assert(initialCount >= 0); + const long maxLong = 0x7fffffff; + m_hSema = CreateSemaphoreW(nullptr, initialCount, maxLong, nullptr); + assert(m_hSema); + } + + AE_NO_TSAN ~Semaphore() + { + CloseHandle(m_hSema); + } + + bool wait() AE_NO_TSAN + { + const unsigned long infinite = 0xffffffff; + return WaitForSingleObject(m_hSema, infinite) == 0; + } + + bool try_wait() AE_NO_TSAN + { + return WaitForSingleObject(m_hSema, 0) == 0; + } + + bool timed_wait(std::uint64_t usecs) AE_NO_TSAN + { + return WaitForSingleObject(m_hSema, (unsigned long)(usecs / 1000)) == 0; + } + + void signal(int count = 1) AE_NO_TSAN + { + while (!ReleaseSemaphore(m_hSema, count, nullptr)); + } + }; +#elif defined(__MACH__) + //--------------------------------------------------------- + // Semaphore (Apple iOS and OSX) + // Can't use POSIX semaphores due to http://lists.apple.com/archives/darwin-kernel/2009/Apr/msg00010.html + //--------------------------------------------------------- + class Semaphore + { + private: + semaphore_t m_sema; + + Semaphore(const Semaphore& other); + Semaphore& operator=(const Semaphore& other); + + public: + AE_NO_TSAN Semaphore(int initialCount = 0) : m_sema() + { + assert(initialCount >= 0); + kern_return_t rc = semaphore_create(mach_task_self(), &m_sema, SYNC_POLICY_FIFO, initialCount); + assert(rc == KERN_SUCCESS); + AE_UNUSED(rc); + } + + AE_NO_TSAN ~Semaphore() + { + semaphore_destroy(mach_task_self(), m_sema); + } + + bool wait() AE_NO_TSAN + { + return semaphore_wait(m_sema) == KERN_SUCCESS; + } + + bool try_wait() AE_NO_TSAN + { + return timed_wait(0); + } + + bool timed_wait(std::uint64_t timeout_usecs) AE_NO_TSAN + { + mach_timespec_t ts; + ts.tv_sec = static_cast(timeout_usecs / 1000000); + ts.tv_nsec = static_cast((timeout_usecs % 1000000) * 1000); + + // added in OSX 10.10: https://developer.apple.com/library/prerelease/mac/documentation/General/Reference/APIDiffsMacOSX10_10SeedDiff/modules/Darwin.html + kern_return_t rc = semaphore_timedwait(m_sema, ts); + return rc == KERN_SUCCESS; + } + + void signal() AE_NO_TSAN + { + while (semaphore_signal(m_sema) != KERN_SUCCESS); + } + + void signal(int count) AE_NO_TSAN + { + while (count-- > 0) + { + while (semaphore_signal(m_sema) != KERN_SUCCESS); + } + } + }; +#elif defined(__unix__) + //--------------------------------------------------------- + // Semaphore (POSIX, Linux) + //--------------------------------------------------------- + class Semaphore + { + private: + sem_t m_sema; + + Semaphore(const Semaphore& other); + Semaphore& operator=(const Semaphore& other); + + public: + AE_NO_TSAN Semaphore(int initialCount = 0) : m_sema() + { + assert(initialCount >= 0); + int rc = sem_init(&m_sema, 0, static_cast(initialCount)); + assert(rc == 0); + AE_UNUSED(rc); + } + + AE_NO_TSAN ~Semaphore() + { + sem_destroy(&m_sema); + } + + bool wait() AE_NO_TSAN + { + // http://stackoverflow.com/questions/2013181/gdb-causes-sem-wait-to-fail-with-eintr-error + int rc; + do + { + rc = sem_wait(&m_sema); + } + while (rc == -1 && errno == EINTR); + return rc == 0; + } + + bool try_wait() AE_NO_TSAN + { + int rc; + do { + rc = sem_trywait(&m_sema); + } while (rc == -1 && errno == EINTR); + return rc == 0; + } + + bool timed_wait(std::uint64_t usecs) AE_NO_TSAN + { + struct timespec ts; + const int usecs_in_1_sec = 1000000; + const int nsecs_in_1_sec = 1000000000; + clock_gettime(CLOCK_REALTIME, &ts); + ts.tv_sec += static_cast(usecs / usecs_in_1_sec); + ts.tv_nsec += static_cast(usecs % usecs_in_1_sec) * 1000; + // sem_timedwait bombs if you have more than 1e9 in tv_nsec + // so we have to clean things up before passing it in + if (ts.tv_nsec >= nsecs_in_1_sec) { + ts.tv_nsec -= nsecs_in_1_sec; + ++ts.tv_sec; + } + + int rc; + do { + rc = sem_timedwait(&m_sema, &ts); + } while (rc == -1 && errno == EINTR); + return rc == 0; + } + + void signal() AE_NO_TSAN + { + while (sem_post(&m_sema) == -1); + } + + void signal(int count) AE_NO_TSAN + { + while (count-- > 0) + { + while (sem_post(&m_sema) == -1); + } + } + }; +#else +#error Unsupported platform! (No semaphore wrapper available) +#endif + + //--------------------------------------------------------- + // LightweightSemaphore + //--------------------------------------------------------- + class LightweightSemaphore + { + public: + typedef std::make_signed::type ssize_t; + + private: + weak_atomic m_count; + Semaphore m_sema; + + bool waitWithPartialSpinning(std::int64_t timeout_usecs = -1) AE_NO_TSAN + { + ssize_t oldCount; + // Is there a better way to set the initial spin count? + // If we lower it to 1000, testBenaphore becomes 15x slower on my Core i7-5930K Windows PC, + // as threads start hitting the kernel semaphore. + int spin = 1024; + while (--spin >= 0) + { + if (m_count.load() > 0) + { + m_count.fetch_add_acquire(-1); + return true; + } + compiler_fence(memory_order_acquire); // Prevent the compiler from collapsing the loop. + } + oldCount = m_count.fetch_add_acquire(-1); + if (oldCount > 0) + return true; + if (timeout_usecs < 0) + { + if (m_sema.wait()) + return true; + } + if (timeout_usecs > 0 && m_sema.timed_wait(static_cast(timeout_usecs))) + return true; + // At this point, we've timed out waiting for the semaphore, but the + // count is still decremented indicating we may still be waiting on + // it. So we have to re-adjust the count, but only if the semaphore + // wasn't signaled enough times for us too since then. If it was, we + // need to release the semaphore too. + while (true) + { + oldCount = m_count.fetch_add_release(1); + if (oldCount < 0) + return false; // successfully restored things to the way they were + // Oh, the producer thread just signaled the semaphore after all. Try again: + oldCount = m_count.fetch_add_acquire(-1); + if (oldCount > 0 && m_sema.try_wait()) + return true; + } + } + + public: + AE_NO_TSAN LightweightSemaphore(ssize_t initialCount = 0) : m_count(initialCount), m_sema() + { + assert(initialCount >= 0); + } + + bool tryWait() AE_NO_TSAN + { + if (m_count.load() > 0) + { + m_count.fetch_add_acquire(-1); + return true; + } + return false; + } + + bool wait() AE_NO_TSAN + { + return tryWait() || waitWithPartialSpinning(); + } + + bool wait(std::int64_t timeout_usecs) AE_NO_TSAN + { + return tryWait() || waitWithPartialSpinning(timeout_usecs); + } + + void signal(ssize_t count = 1) AE_NO_TSAN + { + assert(count >= 0); + ssize_t oldCount = m_count.fetch_add_release(count); + assert(oldCount >= -1); + if (oldCount < 0) + { + m_sema.signal(1); + } + } + + std::size_t availableApprox() const AE_NO_TSAN + { + ssize_t count = m_count.load(); + return count > 0 ? static_cast(count) : 0; + } + }; + } // end namespace spsc_sema +} // end namespace moodycamel + +#if defined(AE_VCPP) && (_MSC_VER < 1700 || defined(__cplusplus_cli)) +#pragma warning(pop) +#ifdef __cplusplus_cli +#pragma managed(pop) +#endif +#endif diff --git a/moodycamel/blockingconcurrentqueue.h b/moodycamel/blockingconcurrentqueue.h old mode 100644 new mode 100755 index c855f9df9..66579b6ca --- a/moodycamel/blockingconcurrentqueue.h +++ b/moodycamel/blockingconcurrentqueue.h @@ -1,419 +1,23 @@ // Provides an efficient blocking version of moodycamel::ConcurrentQueue. -// ©2015-2016 Cameron Desrochers. Distributed under the terms of the simplified +// ©2015-2020 Cameron Desrochers. Distributed under the terms of the simplified // BSD license, available at the top of concurrentqueue.h. +// Also dual-licensed under the Boost Software License (see LICENSE.md) // Uses Jeff Preshing's semaphore implementation (under the terms of its -// separate zlib license, embedded below). +// separate zlib license, see lightweightsemaphore.h). #pragma once #include "concurrentqueue.h" +#include "lightweightsemaphore.h" + #include #include #include #include #include -#if defined(_WIN32) -// Avoid including windows.h in a header; we only need a handful of -// items, so we'll redeclare them here (this is relatively safe since -// the API generally has to remain stable between Windows versions). -// I know this is an ugly hack but it still beats polluting the global -// namespace with thousands of generic names or adding a .cpp for nothing. -extern "C" { - struct _SECURITY_ATTRIBUTES; - __declspec(dllimport) void* __stdcall CreateSemaphoreW(_SECURITY_ATTRIBUTES* lpSemaphoreAttributes, long lInitialCount, long lMaximumCount, const wchar_t* lpName); - __declspec(dllimport) int __stdcall CloseHandle(void* hObject); - __declspec(dllimport) unsigned long __stdcall WaitForSingleObject(void* hHandle, unsigned long dwMilliseconds); - __declspec(dllimport) int __stdcall ReleaseSemaphore(void* hSemaphore, long lReleaseCount, long* lpPreviousCount); -} -#elif defined(__MACH__) -#include -#elif defined(__unix__) -#include -#endif - namespace moodycamel { -namespace details -{ - // Code in the mpmc_sema namespace below is an adaptation of Jeff Preshing's - // portable + lightweight semaphore implementations, originally from - // https://github.com/preshing/cpp11-on-multicore/blob/master/common/sema.h - // LICENSE: - // Copyright (c) 2015 Jeff Preshing - // - // This software is provided 'as-is', without any express or implied - // warranty. In no event will the authors be held liable for any damages - // arising from the use of this software. - // - // Permission is granted to anyone to use this software for any purpose, - // including commercial applications, and to alter it and redistribute it - // freely, subject to the following restrictions: - // - // 1. The origin of this software must not be misrepresented; you must not - // claim that you wrote the original software. If you use this software - // in a product, an acknowledgement in the product documentation would be - // appreciated but is not required. - // 2. Altered source versions must be plainly marked as such, and must not be - // misrepresented as being the original software. - // 3. This notice may not be removed or altered from any source distribution. - namespace mpmc_sema - { -#if defined(_WIN32) - class Semaphore - { - private: - void* m_hSema; - - Semaphore(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; - Semaphore& operator=(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; - - public: - Semaphore(int initialCount = 0) - { - assert(initialCount >= 0); - const long maxLong = 0x7fffffff; - m_hSema = CreateSemaphoreW(nullptr, initialCount, maxLong, nullptr); - } - - ~Semaphore() - { - CloseHandle(m_hSema); - } - - void wait() - { - const unsigned long infinite = 0xffffffff; - WaitForSingleObject(m_hSema, infinite); - } - - bool try_wait() - { - const unsigned long RC_WAIT_TIMEOUT = 0x00000102; - return WaitForSingleObject(m_hSema, 0) != RC_WAIT_TIMEOUT; - } - - bool timed_wait(std::uint64_t usecs) - { - const unsigned long RC_WAIT_TIMEOUT = 0x00000102; - return WaitForSingleObject(m_hSema, (unsigned long)(usecs / 1000)) != RC_WAIT_TIMEOUT; - } - - void signal(int count = 1) - { - ReleaseSemaphore(m_hSema, count, nullptr); - } - }; -#elif defined(__MACH__) - //--------------------------------------------------------- - // Semaphore (Apple iOS and OSX) - // Can't use POSIX semaphores due to http://lists.apple.com/archives/darwin-kernel/2009/Apr/msg00010.html - //--------------------------------------------------------- - class Semaphore - { - private: - semaphore_t m_sema; - - Semaphore(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; - Semaphore& operator=(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; - - public: - Semaphore(int initialCount = 0) - { - assert(initialCount >= 0); - semaphore_create(mach_task_self(), &m_sema, SYNC_POLICY_FIFO, initialCount); - } - - ~Semaphore() - { - semaphore_destroy(mach_task_self(), m_sema); - } - - void wait() - { - semaphore_wait(m_sema); - } - - bool try_wait() - { - return timed_wait(0); - } - - bool timed_wait(std::uint64_t timeout_usecs) - { - mach_timespec_t ts; - ts.tv_sec = static_cast(timeout_usecs / 1000000); - ts.tv_nsec = (timeout_usecs % 1000000) * 1000; - - // added in OSX 10.10: https://developer.apple.com/library/prerelease/mac/documentation/General/Reference/APIDiffsMacOSX10_10SeedDiff/modules/Darwin.html - kern_return_t rc = semaphore_timedwait(m_sema, ts); - - return rc != KERN_OPERATION_TIMED_OUT && rc != KERN_ABORTED; - } - - void signal() - { - semaphore_signal(m_sema); - } - - void signal(int count) - { - while (count-- > 0) - { - semaphore_signal(m_sema); - } - } - }; -#elif defined(__unix__) - //--------------------------------------------------------- - // Semaphore (POSIX, Linux) - //--------------------------------------------------------- - class Semaphore - { - private: - sem_t m_sema; - - Semaphore(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; - Semaphore& operator=(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; - - public: - Semaphore(int initialCount = 0) - { - assert(initialCount >= 0); - sem_init(&m_sema, 0, initialCount); - } - - ~Semaphore() - { - sem_destroy(&m_sema); - } - - void wait() - { - // http://stackoverflow.com/questions/2013181/gdb-causes-sem-wait-to-fail-with-eintr-error - int rc; - do { - rc = sem_wait(&m_sema); - } while (rc == -1 && errno == EINTR); - } - - bool try_wait() - { - int rc; - do { - rc = sem_trywait(&m_sema); - } while (rc == -1 && errno == EINTR); - return !(rc == -1 && errno == EAGAIN); - } - - bool timed_wait(std::uint64_t usecs) - { - struct timespec ts; - const int usecs_in_1_sec = 1000000; - const int nsecs_in_1_sec = 1000000000; - clock_gettime(CLOCK_REALTIME, &ts); - ts.tv_sec += usecs / usecs_in_1_sec; - ts.tv_nsec += (usecs % usecs_in_1_sec) * 1000; - // sem_timedwait bombs if you have more than 1e9 in tv_nsec - // so we have to clean things up before passing it in - if (ts.tv_nsec >= nsecs_in_1_sec) { - ts.tv_nsec -= nsecs_in_1_sec; - ++ts.tv_sec; - } - - int rc; - do { - rc = sem_timedwait(&m_sema, &ts); - } while (rc == -1 && errno == EINTR); - return !(rc == -1 && errno == ETIMEDOUT); - } - - void signal() - { - sem_post(&m_sema); - } - - void signal(int count) - { - while (count-- > 0) - { - sem_post(&m_sema); - } - } - }; -#else -#error Unsupported platform! (No semaphore wrapper available) -#endif - - //--------------------------------------------------------- - // LightweightSemaphore - //--------------------------------------------------------- - class LightweightSemaphore - { - public: - typedef std::make_signed::type ssize_t; - - private: - std::atomic m_count; - Semaphore m_sema; - - bool waitWithPartialSpinning(std::int64_t timeout_usecs = -1) - { - ssize_t oldCount; - // Is there a better way to set the initial spin count? - // If we lower it to 1000, testBenaphore becomes 15x slower on my Core i7-5930K Windows PC, - // as threads start hitting the kernel semaphore. - int spin = 10000; - while (--spin >= 0) - { - oldCount = m_count.load(std::memory_order_relaxed); - if ((oldCount > 0) && m_count.compare_exchange_strong(oldCount, oldCount - 1, std::memory_order_acquire, std::memory_order_relaxed)) - return true; - std::atomic_signal_fence(std::memory_order_acquire); // Prevent the compiler from collapsing the loop. - } - oldCount = m_count.fetch_sub(1, std::memory_order_acquire); - if (oldCount > 0) - return true; - if (timeout_usecs < 0) - { - m_sema.wait(); - return true; - } - if (m_sema.timed_wait((std::uint64_t)timeout_usecs)) - return true; - // At this point, we've timed out waiting for the semaphore, but the - // count is still decremented indicating we may still be waiting on - // it. So we have to re-adjust the count, but only if the semaphore - // wasn't signaled enough times for us too since then. If it was, we - // need to release the semaphore too. - while (true) - { - oldCount = m_count.load(std::memory_order_acquire); - if (oldCount >= 0 && m_sema.try_wait()) - return true; - if (oldCount < 0 && m_count.compare_exchange_strong(oldCount, oldCount + 1, std::memory_order_relaxed, std::memory_order_relaxed)) - return false; - } - } - - ssize_t waitManyWithPartialSpinning(ssize_t max, std::int64_t timeout_usecs = -1) - { - assert(max > 0); - ssize_t oldCount; - int spin = 10000; - while (--spin >= 0) - { - oldCount = m_count.load(std::memory_order_relaxed); - if (oldCount > 0) - { - ssize_t newCount = oldCount > max ? oldCount - max : 0; - if (m_count.compare_exchange_strong(oldCount, newCount, std::memory_order_acquire, std::memory_order_relaxed)) - return oldCount - newCount; - } - std::atomic_signal_fence(std::memory_order_acquire); - } - oldCount = m_count.fetch_sub(1, std::memory_order_acquire); - if (oldCount <= 0) - { - if (timeout_usecs < 0) - m_sema.wait(); - else if (!m_sema.timed_wait((std::uint64_t)timeout_usecs)) - { - while (true) - { - oldCount = m_count.load(std::memory_order_acquire); - if (oldCount >= 0 && m_sema.try_wait()) - break; - if (oldCount < 0 && m_count.compare_exchange_strong(oldCount, oldCount + 1, std::memory_order_relaxed, std::memory_order_relaxed)) - return 0; - } - } - } - if (max > 1) - return 1 + tryWaitMany(max - 1); - return 1; - } - - public: - LightweightSemaphore(ssize_t initialCount = 0) : m_count(initialCount) - { - assert(initialCount >= 0); - } - - bool tryWait() - { - ssize_t oldCount = m_count.load(std::memory_order_relaxed); - while (oldCount > 0) - { - if (m_count.compare_exchange_weak(oldCount, oldCount - 1, std::memory_order_acquire, std::memory_order_relaxed)) - return true; - } - return false; - } - - void wait() - { - if (!tryWait()) - waitWithPartialSpinning(); - } - - bool wait(std::int64_t timeout_usecs) - { - return tryWait() || waitWithPartialSpinning(timeout_usecs); - } - - // Acquires between 0 and (greedily) max, inclusive - ssize_t tryWaitMany(ssize_t max) - { - assert(max >= 0); - ssize_t oldCount = m_count.load(std::memory_order_relaxed); - while (oldCount > 0) - { - ssize_t newCount = oldCount > max ? oldCount - max : 0; - if (m_count.compare_exchange_weak(oldCount, newCount, std::memory_order_acquire, std::memory_order_relaxed)) - return oldCount - newCount; - } - return 0; - } - - // Acquires at least one, and (greedily) at most max - ssize_t waitMany(ssize_t max, std::int64_t timeout_usecs) - { - assert(max >= 0); - ssize_t result = tryWaitMany(max); - if (result == 0 && max > 0) - result = waitManyWithPartialSpinning(max, timeout_usecs); - return result; - } - - ssize_t waitMany(ssize_t max) - { - ssize_t result = waitMany(max, -1); - assert(result > 0); - return result; - } - - void signal(ssize_t count = 1) - { - assert(count >= 0); - ssize_t oldCount = m_count.fetch_add(count, std::memory_order_release); - ssize_t toRelease = -oldCount < count ? -oldCount : count; - if (toRelease > 0) - { - m_sema.signal((int)toRelease); - } - } - - ssize_t availableApprox() const - { - ssize_t count = m_count.load(std::memory_order_relaxed); - return count > 0 ? count : 0; - } - }; - } // end namespace mpmc_sema -} // end namespace details - - // This is a blocking version of the queue. It has an almost identical interface to // the normal non-blocking version, with the addition of various wait_dequeue() methods // and the removal of producer-specific dequeue methods. @@ -422,7 +26,7 @@ class BlockingConcurrentQueue { private: typedef ::moodycamel::ConcurrentQueue ConcurrentQueue; - typedef details::mpmc_sema::LightweightSemaphore LightweightSemaphore; + typedef ::moodycamel::LightweightSemaphore LightweightSemaphore; public: typedef typename ConcurrentQueue::producer_token_t producer_token_t; @@ -452,7 +56,7 @@ class BlockingConcurrentQueue // includes making the memory effects of construction visible, possibly with a // memory barrier). explicit BlockingConcurrentQueue(size_t capacity = 6 * BLOCK_SIZE) - : inner(capacity), sema(create(), &BlockingConcurrentQueue::template destroy) + : inner(capacity), sema(create(0, (int)Traits::MAX_SEMA_SPINS), &BlockingConcurrentQueue::template destroy) { assert(reinterpret_cast((BlockingConcurrentQueue*)1) == &((BlockingConcurrentQueue*)1)->inner && "BlockingConcurrentQueue must have ConcurrentQueue as its first member"); if (!sema) { @@ -461,7 +65,7 @@ class BlockingConcurrentQueue } BlockingConcurrentQueue(size_t minCapacity, size_t maxExplicitProducers, size_t maxImplicitProducers) - : inner(minCapacity, maxExplicitProducers, maxImplicitProducers), sema(create(), &BlockingConcurrentQueue::template destroy) + : inner(minCapacity, maxExplicitProducers, maxImplicitProducers), sema(create(0, (int)Traits::MAX_SEMA_SPINS), &BlockingConcurrentQueue::template destroy) { assert(reinterpret_cast((BlockingConcurrentQueue*)1) == &((BlockingConcurrentQueue*)1)->inner && "BlockingConcurrentQueue must have ConcurrentQueue as its first member"); if (!sema) { @@ -754,7 +358,9 @@ class BlockingConcurrentQueue template inline void wait_dequeue(U& item) { - sema->wait(); + while (!sema->wait()) { + continue; + } while (!inner.try_dequeue(item)) { continue; } @@ -795,7 +401,9 @@ class BlockingConcurrentQueue template inline void wait_dequeue(consumer_token_t& token, U& item) { - sema->wait(); + while (!sema->wait()) { + continue; + } while (!inner.try_dequeue(token, item)) { continue; } @@ -943,18 +551,11 @@ class BlockingConcurrentQueue private: - template - static inline U* create() - { - auto p = (Traits::malloc)(sizeof(U)); - return p != nullptr ? new (p) U : nullptr; - } - - template - static inline U* create(A1&& a1) + template + static inline U* create(A1&& a1, A2&& a2) { - auto p = (Traits::malloc)(sizeof(U)); - return p != nullptr ? new (p) U(std::forward(a1)) : nullptr; + void* p = (Traits::malloc)(sizeof(U)); + return p != nullptr ? new (p) U(std::forward(a1), std::forward(a2)) : nullptr; } template diff --git a/moodycamel/concurrentqueue.h b/moodycamel/concurrentqueue.h old mode 100644 new mode 100755 index aa9a10cde..609ca4ab5 --- a/moodycamel/concurrentqueue.h +++ b/moodycamel/concurrentqueue.h @@ -1,11 +1,11 @@ -// Provides a C++11 implementation of a multi-producer, multi-consumer lock-free queue. +// Provides a C++11 implementation of a multi-producer, multi-consumer lock-free queue. // An overview, including benchmark results, is provided here: // http://moodycamel.com/blog/2014/a-fast-general-purpose-lock-free-queue-for-c++ // The full design is also described in excruciating detail at: // http://moodycamel.com/blog/2014/detailed-design-of-a-lock-free-queue // Simplified BSD license: -// Copyright (c) 2013-2016, Cameron Desrochers. +// Copyright (c) 2013-2020, Cameron Desrochers. // All rights reserved. // // Redistribution and use in source and binary forms, with or without modification, @@ -27,10 +27,11 @@ // TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, // EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +// Also dual-licensed under the Boost Software License (see LICENSE.md) #pragma once -#if defined(__GNUC__) +#if defined(__GNUC__) && !defined(__INTEL_COMPILER) // Disable -Wconversion warnings (spuriously triggered when Traits::size_t and // Traits::index_t are set to < 32 bits, causing integer promotion, causing warnings // upon assigning any computed values) @@ -42,6 +43,13 @@ #endif #endif +#if defined(_MSC_VER) && (!defined(_HAS_CXX17) || !_HAS_CXX17) +// VS2019 with /W4 warns about constant conditional expressions but unless /std=c++17 or higher +// does not support `if constexpr`, so we have no choice but to simply disable the warning +#pragma warning(push) +#pragma warning(disable: 4127) // conditional expression is constant +#endif + #if defined(__APPLE__) #include "TargetConditionals.h" #endif @@ -96,7 +104,7 @@ namespace moodycamel { namespace details { static const thread_id_t invalid_thread_id2 = 0xFFFFFFFFU; // Not technically guaranteed to be invalid, but is never used in practice. Note that all Win32 thread IDs are presently multiples of 4. static inline thread_id_t thread_id() { return static_cast(::GetCurrentThreadId()); } } } -#elif defined(__arm__) || defined(_M_ARM) || defined(__aarch64__) || (defined(__APPLE__) && TARGET_OS_IPHONE) +#elif defined(__arm__) || defined(_M_ARM) || defined(__aarch64__) || (defined(__APPLE__) && TARGET_OS_IPHONE) || defined(MOODYCAMEL_NO_THREAD_LOCAL) namespace moodycamel { namespace details { static_assert(sizeof(std::thread::id) == 4 || sizeof(std::thread::id) == 8, "std::thread::id is expected to be either 4 or 8 bytes"); @@ -146,10 +154,21 @@ namespace moodycamel { namespace details { typedef std::uintptr_t thread_id_t; static const thread_id_t invalid_thread_id = 0; // Address can't be nullptr static const thread_id_t invalid_thread_id2 = 1; // Member accesses off a null pointer are also generally invalid. Plus it's not aligned. - static inline thread_id_t thread_id() { static MOODYCAMEL_THREADLOCAL int x; return reinterpret_cast(&x); } + inline thread_id_t thread_id() { static MOODYCAMEL_THREADLOCAL int x; return reinterpret_cast(&x); } } } #endif +// Constexpr if +#ifndef MOODYCAMEL_CONSTEXPR_IF +#if (defined(_MSC_VER) && defined(_HAS_CXX17) && _HAS_CXX17) || __cplusplus > 201402L +#define MOODYCAMEL_CONSTEXPR_IF if constexpr +#define MOODYCAMEL_MAYBE_UNUSED [[maybe_unused]] +#else +#define MOODYCAMEL_CONSTEXPR_IF if +#define MOODYCAMEL_MAYBE_UNUSED +#endif +#endif + // Exceptions #ifndef MOODYCAMEL_EXCEPTIONS_ENABLED #if (defined(_MSC_VER) && defined(_CPPUNWIND)) || (defined(__GNUC__) && defined(__EXCEPTIONS)) || (!defined(_MSC_VER) && !defined(__GNUC__)) @@ -162,8 +181,8 @@ namespace moodycamel { namespace details { #define MOODYCAMEL_RETHROW throw #define MOODYCAMEL_THROW(expr) throw (expr) #else -#define MOODYCAMEL_TRY if (true) -#define MOODYCAMEL_CATCH(...) else if (false) +#define MOODYCAMEL_TRY MOODYCAMEL_CONSTEXPR_IF (true) +#define MOODYCAMEL_CATCH(...) else MOODYCAMEL_CONSTEXPR_IF (false) #define MOODYCAMEL_RETHROW #define MOODYCAMEL_THROW(expr) #endif @@ -214,6 +233,44 @@ namespace moodycamel { namespace details { #endif #endif +namespace moodycamel { namespace details { +#ifndef MOODYCAMEL_ALIGNAS +// VS2013 doesn't support alignas or alignof, and align() requires a constant literal +#if defined(_MSC_VER) && _MSC_VER <= 1800 +#define MOODYCAMEL_ALIGNAS(alignment) __declspec(align(alignment)) +#define MOODYCAMEL_ALIGNOF(obj) __alignof(obj) +#define MOODYCAMEL_ALIGNED_TYPE_LIKE(T, obj) typename details::Vs2013Aligned::value, T>::type + template struct Vs2013Aligned { }; // default, unsupported alignment + template struct Vs2013Aligned<1, T> { typedef __declspec(align(1)) T type; }; + template struct Vs2013Aligned<2, T> { typedef __declspec(align(2)) T type; }; + template struct Vs2013Aligned<4, T> { typedef __declspec(align(4)) T type; }; + template struct Vs2013Aligned<8, T> { typedef __declspec(align(8)) T type; }; + template struct Vs2013Aligned<16, T> { typedef __declspec(align(16)) T type; }; + template struct Vs2013Aligned<32, T> { typedef __declspec(align(32)) T type; }; + template struct Vs2013Aligned<64, T> { typedef __declspec(align(64)) T type; }; + template struct Vs2013Aligned<128, T> { typedef __declspec(align(128)) T type; }; + template struct Vs2013Aligned<256, T> { typedef __declspec(align(256)) T type; }; +#else + template struct identity { typedef T type; }; +#define MOODYCAMEL_ALIGNAS(alignment) alignas(alignment) +#define MOODYCAMEL_ALIGNOF(obj) alignof(obj) +#define MOODYCAMEL_ALIGNED_TYPE_LIKE(T, obj) alignas(alignof(obj)) typename details::identity::type +#endif +#endif +} } + + +// TSAN can false report races in lock-free code. To enable TSAN to be used from projects that use this one, +// we can apply per-function compile-time suppression. +// See https://clang.llvm.org/docs/ThreadSanitizer.html#has-feature-thread-sanitizer +#define MOODYCAMEL_NO_TSAN +#if defined(__has_feature) + #if __has_feature(thread_sanitizer) + #undef MOODYCAMEL_NO_TSAN + #define MOODYCAMEL_NO_TSAN __attribute__((no_sanitize("thread"))) + #endif // TSAN +#endif // TSAN + // Compiler-specific likely/unlikely hints namespace moodycamel { namespace details { #if defined(__GNUC__) @@ -315,6 +372,12 @@ struct ConcurrentQueueDefaultTraits // that this limit is enforced at the block level (for performance reasons), i.e. // it's rounded up to the nearest block size. static const size_t MAX_SUBQUEUE_SIZE = details::const_numeric_max::value; + + // The number of times to spin before sleeping when waiting on a semaphore. + // Recommended values are on the order of 1000-10000 unless the number of + // consumer threads exceeds the number of idle cores (in which case try 0-100). + // Only affects instances of the BlockingConcurrentQueue. + static const int MAX_SEMA_SPINS = 10000; #ifndef MCDBGQ_USE_RELACY @@ -785,7 +848,7 @@ class ConcurrentQueue } // Destroy implicit producer hash tables - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE != 0) { + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE != 0) { auto hash = implicitProducerHash.load(std::memory_order_relaxed); while (hash != nullptr) { auto prev = hash->prev; @@ -910,8 +973,8 @@ class ConcurrentQueue // Thread-safe. inline bool enqueue(T const& item) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; - return inner_enqueue(item); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; + else return inner_enqueue(item); } // Enqueues a single item (by moving it, if possible). @@ -921,8 +984,8 @@ class ConcurrentQueue // Thread-safe. inline bool enqueue(T&& item) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; - return inner_enqueue(std::move(item)); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; + else return inner_enqueue(std::move(item)); } // Enqueues a single item (by copying it) using an explicit producer token. @@ -952,8 +1015,8 @@ class ConcurrentQueue template bool enqueue_bulk(It itemFirst, size_t count) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; - return inner_enqueue_bulk(itemFirst, count); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; + else return inner_enqueue_bulk(itemFirst, count); } // Enqueues several items using an explicit producer token. @@ -975,8 +1038,8 @@ class ConcurrentQueue // Thread-safe. inline bool try_enqueue(T const& item) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; - return inner_enqueue(item); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; + else return inner_enqueue(item); } // Enqueues a single item (by moving it, if possible). @@ -986,8 +1049,8 @@ class ConcurrentQueue // Thread-safe. inline bool try_enqueue(T&& item) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; - return inner_enqueue(std::move(item)); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; + else return inner_enqueue(std::move(item)); } // Enqueues a single item (by copying it) using an explicit producer token. @@ -1016,8 +1079,8 @@ class ConcurrentQueue template bool try_enqueue_bulk(It itemFirst, size_t count) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; - return inner_enqueue_bulk(itemFirst, count); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return false; + else return inner_enqueue_bulk(itemFirst, count); } // Enqueues several items using an explicit producer token. @@ -1266,7 +1329,10 @@ class ConcurrentQueue private: friend struct ProducerToken; friend struct ConsumerToken; + struct ExplicitProducer; friend struct ExplicitProducer; + struct ImplicitProducer; + friend struct ImplicitProducer; friend class ConcurrentQueueTests; enum AllocationMode { CanAlloc, CannotAlloc }; @@ -1371,7 +1437,7 @@ class ConcurrentQueue inline void add(N* node) { -#if MCDBGQ_NOLOCKFREE_FREELIST +#ifdef MCDBGQ_NOLOCKFREE_FREELIST debug::DebugLock lock(mutex); #endif // We know that the should-be-on-freelist bit is 0 at this point, so it's safe to @@ -1385,7 +1451,7 @@ class ConcurrentQueue inline N* try_get() { -#if MCDBGQ_NOLOCKFREE_FREELIST +#ifdef MCDBGQ_NOLOCKFREE_FREELIST debug::DebugLock lock(mutex); #endif auto head = freeListHead.load(std::memory_order_acquire); @@ -1457,7 +1523,7 @@ class ConcurrentQueue static const std::uint32_t REFS_MASK = 0x7FFFFFFF; static const std::uint32_t SHOULD_BE_ON_FREELIST = 0x80000000; -#if MCDBGQ_NOLOCKFREE_FREELIST +#ifdef MCDBGQ_NOLOCKFREE_FREELIST debug::DebugMutex mutex; #endif }; @@ -1474,7 +1540,7 @@ class ConcurrentQueue Block() : next(nullptr), elementsCompletelyDequeued(0), freeListRefs(0), freeListNext(nullptr), shouldBeOnFreeList(false), dynamicallyAllocated(true) { -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM owner = nullptr; #endif } @@ -1482,7 +1548,7 @@ class ConcurrentQueue template inline bool is_empty() const { - if (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { + MOODYCAMEL_CONSTEXPR_IF (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { // Check flags for (size_t i = 0; i < BLOCK_SIZE; ++i) { if (!emptyFlags[i].load(std::memory_order_relaxed)) { @@ -1507,9 +1573,9 @@ class ConcurrentQueue // Returns true if the block is now empty (does not apply in explicit context) template - inline bool set_empty(index_t i) + inline bool set_empty(MOODYCAMEL_MAYBE_UNUSED index_t i) { - if (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { + MOODYCAMEL_CONSTEXPR_IF (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { // Set flag assert(!emptyFlags[BLOCK_SIZE - 1 - static_cast(i & static_cast(BLOCK_SIZE - 1))].load(std::memory_order_relaxed)); emptyFlags[BLOCK_SIZE - 1 - static_cast(i & static_cast(BLOCK_SIZE - 1))].store(true, std::memory_order_release); @@ -1526,9 +1592,9 @@ class ConcurrentQueue // Sets multiple contiguous item statuses to 'empty' (assumes no wrapping and count > 0). // Returns true if the block is now empty (does not apply in explicit context). template - inline bool set_many_empty(index_t i, size_t count) + inline bool set_many_empty(MOODYCAMEL_MAYBE_UNUSED index_t i, size_t count) { - if (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { + MOODYCAMEL_CONSTEXPR_IF (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { // Set flags std::atomic_thread_fence(std::memory_order_release); i = BLOCK_SIZE - 1 - static_cast(i & static_cast(BLOCK_SIZE - 1)) - count + 1; @@ -1549,7 +1615,7 @@ class ConcurrentQueue template inline void set_all_empty() { - if (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { + MOODYCAMEL_CONSTEXPR_IF (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { // Set all flags for (size_t i = 0; i != BLOCK_SIZE; ++i) { emptyFlags[i].store(true, std::memory_order_relaxed); @@ -1564,7 +1630,7 @@ class ConcurrentQueue template inline void reset_empty() { - if (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { + MOODYCAMEL_CONSTEXPR_IF (context == explicit_context && BLOCK_SIZE <= EXPLICIT_BLOCK_EMPTY_COUNTER_THRESHOLD) { // Reset flags for (size_t i = 0; i != BLOCK_SIZE; ++i) { emptyFlags[i].store(false, std::memory_order_relaxed); @@ -1580,20 +1646,8 @@ class ConcurrentQueue inline T const* operator[](index_t idx) const MOODYCAMEL_NOEXCEPT { return static_cast(static_cast(elements)) + static_cast(idx & static_cast(BLOCK_SIZE - 1)); } private: - // IMPORTANT: This must be the first member in Block, so that if T depends on the alignment of - // addresses returned by malloc, that alignment will be preserved. Apparently clang actually - // generates code that uses this assumption for AVX instructions in some cases. Ideally, we - // should also align Block to the alignment of T in case it's higher than malloc's 16-byte - // alignment, but this is hard to do in a cross-platform way. Assert for this case: - static_assert(std::alignment_of::value <= std::alignment_of::value, "The queue does not support super-aligned types at this time"); - // Additionally, we need the alignment of Block itself to be a multiple of max_align_t since - // otherwise the appropriate padding will not be added at the end of Block in order to make - // arrays of Blocks all be properly aligned (not just the first one). We use a union to force - // this. - union { - char elements[sizeof(T) * BLOCK_SIZE]; - details::max_align_t dummy; - }; + static_assert(std::alignment_of::value <= sizeof(T), "The queue does not support types with an alignment greater than their size at this time"); + MOODYCAMEL_ALIGNED_TYPE_LIKE(char[sizeof(T) * BLOCK_SIZE], T) elements; public: Block* next; std::atomic elementsCompletelyDequeued; @@ -1604,14 +1658,14 @@ class ConcurrentQueue std::atomic shouldBeOnFreeList; bool dynamicallyAllocated; // Perhaps a better name for this would be 'isNotPartOfInitialBlockPool' -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM void* owner; #endif }; - static_assert(std::alignment_of::value >= std::alignment_of::value, "Internal error: Blocks must be at least as aligned as the type they are wrapping"); + static_assert(std::alignment_of::value >= std::alignment_of::value, "Internal error: Blocks must be at least as aligned as the type they are wrapping"); -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM public: struct MemStats; private: @@ -1634,7 +1688,7 @@ class ConcurrentQueue { } - virtual ~ProducerBase() { }; + virtual ~ProducerBase() { } template inline bool dequeue(U& element) @@ -1682,7 +1736,7 @@ class ConcurrentQueue ConcurrentQueue* parent; protected: -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM friend struct MemStats; #endif }; @@ -1694,8 +1748,8 @@ class ConcurrentQueue struct ExplicitProducer : public ProducerBase { - explicit ExplicitProducer(ConcurrentQueue* parent) : - ProducerBase(parent, true), + explicit ExplicitProducer(ConcurrentQueue* parent_) : + ProducerBase(parent_, true), blockIndex(nullptr), pr_blockIndexSlotsUsed(0), pr_blockIndexSize(EXPLICIT_INITIAL_INDEX_SIZE >> 1), @@ -1703,7 +1757,7 @@ class ConcurrentQueue pr_blockIndexEntries(nullptr), pr_blockIndexRaw(nullptr) { - size_t poolBasedIndexSize = details::ceil_to_pow_2(parent->initialBlockPoolSize) >> 1; + size_t poolBasedIndexSize = details::ceil_to_pow_2(parent_->initialBlockPoolSize) >> 1; if (poolBasedIndexSize > pr_blockIndexSize) { pr_blockIndexSize = poolBasedIndexSize; } @@ -1815,7 +1869,10 @@ class ConcurrentQueue // to allocate a new index. Note pr_blockIndexRaw can only be nullptr if // the initial allocation failed in the constructor. - if (allocMode == CannotAlloc || !new_block_index(pr_blockIndexSlotsUsed)) { + MOODYCAMEL_CONSTEXPR_IF (allocMode == CannotAlloc) { + return false; + } + else if (!new_block_index(pr_blockIndexSlotsUsed)) { return false; } } @@ -1825,7 +1882,7 @@ class ConcurrentQueue if (newBlock == nullptr) { return false; } -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM newBlock->owner = this; #endif newBlock->ConcurrentQueue::Block::template reset_empty(); @@ -1839,8 +1896,8 @@ class ConcurrentQueue this->tailBlock = newBlock; ++pr_blockIndexSlotsUsed; } - - if (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (nullptr) T(std::forward(element)))) { + + MOODYCAMEL_CONSTEXPR_IF (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (static_cast(nullptr)) T(std::forward(element)))) { // The constructor may throw. We want the element not to appear in the queue in // that case (without corrupting the queue): MOODYCAMEL_TRY { @@ -1866,7 +1923,7 @@ class ConcurrentQueue blockIndex.load(std::memory_order_relaxed)->front.store(pr_blockIndexFront, std::memory_order_release); pr_blockIndexFront = (pr_blockIndexFront + 1) & (pr_blockIndexSize - 1); - if (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (nullptr) T(std::forward(element)))) { + MOODYCAMEL_CONSTEXPR_IF (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (static_cast(nullptr)) T(std::forward(element)))) { this->tailIndex.store(newTailIndex, std::memory_order_release); return true; } @@ -1910,7 +1967,8 @@ class ConcurrentQueue // incremented after dequeueOptimisticCount -- this is enforced in the `else` block below), and since we now // have a version of dequeueOptimisticCount that is at least as recent as overcommit (due to the release upon // incrementing dequeueOvercommit and the acquire above that synchronizes with it), overcommit <= myDequeueCount. - assert(overcommit <= myDequeueCount); + // However, we can't assert this since both dequeueOptimisticCount and dequeueOvercommit may (independently) + // overflow; in such a case, though, the logic still holds since the difference between the two is maintained. // Note that we reload tail here in case it changed; it will be the same value as before or greater, since // this load is sequenced after (happens after) the earlier load above. This is supported by read-read @@ -1958,12 +2016,12 @@ class ConcurrentQueue block->ConcurrentQueue::Block::template set_empty(index); } } guard = { block, index }; - - element = std::move(el); + + element = std::move(el); // NOLINT } else { - element = std::move(el); - el.~T(); + element = std::move(el); // NOLINT + el.~T(); // NOLINT block->ConcurrentQueue::Block::template set_empty(index); } @@ -1979,7 +2037,7 @@ class ConcurrentQueue } template - bool enqueue_bulk(It itemFirst, size_t count) + bool MOODYCAMEL_NO_TSAN enqueue_bulk(It itemFirst, size_t count) { // First, we need to make sure we have enough room to enqueue all of the elements; // this means pre-allocating blocks and putting them in the block index (but only if @@ -2018,7 +2076,14 @@ class ConcurrentQueue assert(!details::circular_less_than(currentTailIndex, head)); bool full = !details::circular_less_than(head, currentTailIndex + BLOCK_SIZE) || (MAX_SUBQUEUE_SIZE != details::const_numeric_max::value && (MAX_SUBQUEUE_SIZE == 0 || MAX_SUBQUEUE_SIZE - BLOCK_SIZE < currentTailIndex - head)); if (pr_blockIndexRaw == nullptr || pr_blockIndexSlotsUsed == pr_blockIndexSize || full) { - if (allocMode == CannotAlloc || full || !new_block_index(originalBlockIndexSlotsUsed)) { + MOODYCAMEL_CONSTEXPR_IF (allocMode == CannotAlloc) { + // Failed to allocate, undo changes (but keep injected blocks) + pr_blockIndexFront = originalBlockIndexFront; + pr_blockIndexSlotsUsed = originalBlockIndexSlotsUsed; + this->tailBlock = startBlock == nullptr ? firstAllocatedBlock : startBlock; + return false; + } + else if (full || !new_block_index(originalBlockIndexSlotsUsed)) { // Failed to allocate, undo changes (but keep injected blocks) pr_blockIndexFront = originalBlockIndexFront; pr_blockIndexSlotsUsed = originalBlockIndexSlotsUsed; @@ -2041,7 +2106,7 @@ class ConcurrentQueue return false; } -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM newBlock->owner = this; #endif newBlock->ConcurrentQueue::Block::template set_all_empty(); @@ -2074,7 +2139,7 @@ class ConcurrentQueue block = block->next; } - if (MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (nullptr) T(details::deref_noexcept(itemFirst)))) { + MOODYCAMEL_CONSTEXPR_IF (MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (static_cast(nullptr)) T(details::deref_noexcept(itemFirst)))) { blockIndex.load(std::memory_order_relaxed)->front.store((pr_blockIndexFront - 1) & (pr_blockIndexSize - 1), std::memory_order_release); } } @@ -2089,11 +2154,11 @@ class ConcurrentQueue this->tailBlock = firstAllocatedBlock; } while (true) { - auto stopIndex = (currentTailIndex & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); + index_t stopIndex = (currentTailIndex & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); if (details::circular_less_than(newTailIndex, stopIndex)) { stopIndex = newTailIndex; } - if (MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (nullptr) T(details::deref_noexcept(itemFirst)))) { + MOODYCAMEL_CONSTEXPR_IF (MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (static_cast(nullptr)) T(details::deref_noexcept(itemFirst)))) { while (currentTailIndex != stopIndex) { new ((*this->tailBlock)[currentTailIndex++]) T(*itemFirst++); } @@ -2108,7 +2173,7 @@ class ConcurrentQueue // may only define a (noexcept) move constructor, and so calls to the // cctor will not compile, even if they are in an if branch that will never // be executed - new ((*this->tailBlock)[currentTailIndex]) T(details::nomove_if<(bool)!MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (nullptr) T(details::deref_noexcept(itemFirst)))>::eval(*itemFirst)); + new ((*this->tailBlock)[currentTailIndex]) T(details::nomove_if(nullptr)) T(details::deref_noexcept(itemFirst)))>::eval(*itemFirst)); ++currentTailIndex; ++itemFirst; } @@ -2155,8 +2220,9 @@ class ConcurrentQueue this->tailBlock = this->tailBlock->next; } - if (!MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (nullptr) T(details::deref_noexcept(itemFirst))) && firstAllocatedBlock != nullptr) { - blockIndex.load(std::memory_order_relaxed)->front.store((pr_blockIndexFront - 1) & (pr_blockIndexSize - 1), std::memory_order_release); + MOODYCAMEL_CONSTEXPR_IF (!MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (static_cast(nullptr)) T(details::deref_noexcept(itemFirst)))) { + if (firstAllocatedBlock != nullptr) + blockIndex.load(std::memory_order_relaxed)->front.store((pr_blockIndexFront - 1) & (pr_blockIndexSize - 1), std::memory_order_release); } this->tailIndex.store(newTailIndex, std::memory_order_release); @@ -2174,7 +2240,6 @@ class ConcurrentQueue std::atomic_thread_fence(std::memory_order_acquire); auto myDequeueCount = this->dequeueOptimisticCount.fetch_add(desiredCount, std::memory_order_relaxed); - assert(overcommit <= myDequeueCount); tail = this->tailIndex.load(std::memory_order_acquire); auto actualCount = static_cast(tail - (myDequeueCount - overcommit)); @@ -2201,7 +2266,7 @@ class ConcurrentQueue auto index = firstIndex; do { auto firstIndexInBlock = index; - auto endIndex = (index & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); + index_t endIndex = (index & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); endIndex = details::circular_less_than(firstIndex + static_cast(actualCount), endIndex) ? firstIndex + static_cast(actualCount) : endIndex; auto block = localBlockIndex->entries[indexIndex].block; if (MOODYCAMEL_NOEXCEPT_ASSIGN(T, T&&, details::deref_noexcept(itemFirst) = std::move((*(*block)[index])))) { @@ -2328,7 +2393,7 @@ class ConcurrentQueue private: #endif -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM friend struct MemStats; #endif }; @@ -2340,8 +2405,8 @@ class ConcurrentQueue struct ImplicitProducer : public ProducerBase { - ImplicitProducer(ConcurrentQueue* parent) : - ProducerBase(parent, false), + ImplicitProducer(ConcurrentQueue* parent_) : + ProducerBase(parent_, false), nextBlockIndexCapacity(IMPLICIT_INITIAL_INDEX_SIZE), blockIndex(nullptr) { @@ -2415,7 +2480,7 @@ class ConcurrentQueue if (!details::circular_less_than(head, currentTailIndex + BLOCK_SIZE) || (MAX_SUBQUEUE_SIZE != details::const_numeric_max::value && (MAX_SUBQUEUE_SIZE == 0 || MAX_SUBQUEUE_SIZE - BLOCK_SIZE < currentTailIndex - head))) { return false; } -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX debug::DebugLock lock(mutex); #endif // Find out where we'll be inserting this block in the block index @@ -2431,12 +2496,12 @@ class ConcurrentQueue idxEntry->value.store(nullptr, std::memory_order_relaxed); return false; } -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM newBlock->owner = this; #endif newBlock->ConcurrentQueue::Block::template reset_empty(); - - if (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (nullptr) T(std::forward(element)))) { + + MOODYCAMEL_CONSTEXPR_IF (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (static_cast(nullptr)) T(std::forward(element)))) { // May throw, try to insert now before we publish the fact that we have this new block MOODYCAMEL_TRY { new ((*newBlock)[currentTailIndex]) T(std::forward(element)); @@ -2454,7 +2519,7 @@ class ConcurrentQueue this->tailBlock = newBlock; - if (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (nullptr) T(std::forward(element)))) { + MOODYCAMEL_CONSTEXPR_IF (!MOODYCAMEL_NOEXCEPT_CTOR(T, U, new (static_cast(nullptr)) T(std::forward(element)))) { this->tailIndex.store(newTailIndex, std::memory_order_release); return true; } @@ -2477,7 +2542,6 @@ class ConcurrentQueue std::atomic_thread_fence(std::memory_order_acquire); index_t myDequeueCount = this->dequeueOptimisticCount.fetch_add(1, std::memory_order_relaxed); - assert(overcommit <= myDequeueCount); tail = this->tailIndex.load(std::memory_order_acquire); if ((details::likely)(details::circular_less_than(myDequeueCount - overcommit, tail))) { index_t index = this->headIndex.fetch_add(1, std::memory_order_acq_rel); @@ -2490,7 +2554,7 @@ class ConcurrentQueue auto& el = *((*block)[index]); if (!MOODYCAMEL_NOEXCEPT_ASSIGN(T, T&&, element = std::move(el))) { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX // Note: Acquiring the mutex with every dequeue instead of only when a block // is released is very sub-optimal, but it is, after all, purely debug code. debug::DebugLock lock(producer->mutex); @@ -2510,16 +2574,16 @@ class ConcurrentQueue } } } guard = { block, index, entry, this->parent }; - - element = std::move(el); + + element = std::move(el); // NOLINT } else { - element = std::move(el); - el.~T(); - + element = std::move(el); // NOLINT + el.~T(); // NOLINT + if (block->ConcurrentQueue::Block::template set_empty(index)) { { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX debug::DebugLock lock(mutex); #endif // Add the block back into the global free pool (and remove from block index) @@ -2539,6 +2603,10 @@ class ConcurrentQueue return false; } +#ifdef _MSC_VER +#pragma warning(push) +#pragma warning(disable: 4706) // assignment within conditional expression +#endif template bool enqueue_bulk(It itemFirst, size_t count) { @@ -2560,7 +2628,7 @@ class ConcurrentQueue size_t blockBaseDiff = ((startTailIndex + count - 1) & ~static_cast(BLOCK_SIZE - 1)) - ((startTailIndex - 1) & ~static_cast(BLOCK_SIZE - 1)); index_t currentTailIndex = (startTailIndex - 1) & ~static_cast(BLOCK_SIZE - 1); if (blockBaseDiff > 0) { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX debug::DebugLock lock(mutex); #endif do { @@ -2574,6 +2642,7 @@ class ConcurrentQueue auto head = this->headIndex.load(std::memory_order_relaxed); assert(!details::circular_less_than(currentTailIndex, head)); bool full = !details::circular_less_than(head, currentTailIndex + BLOCK_SIZE) || (MAX_SUBQUEUE_SIZE != details::const_numeric_max::value && (MAX_SUBQUEUE_SIZE == 0 || MAX_SUBQUEUE_SIZE - BLOCK_SIZE < currentTailIndex - head)); + if (full || !(indexInserted = insert_block_index_entry(idxEntry, currentTailIndex)) || (newBlock = this->parent->ConcurrentQueue::template requisition_block()) == nullptr) { // Index allocation or block allocation failed; revert any other allocations // and index insertions done so far for this operation @@ -2594,7 +2663,7 @@ class ConcurrentQueue return false; } -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM newBlock->owner = this; #endif newBlock->ConcurrentQueue::Block::template reset_empty(); @@ -2624,11 +2693,11 @@ class ConcurrentQueue this->tailBlock = firstAllocatedBlock; } while (true) { - auto stopIndex = (currentTailIndex & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); + index_t stopIndex = (currentTailIndex & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); if (details::circular_less_than(newTailIndex, stopIndex)) { stopIndex = newTailIndex; } - if (MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (nullptr) T(details::deref_noexcept(itemFirst)))) { + MOODYCAMEL_CONSTEXPR_IF (MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (static_cast(nullptr)) T(details::deref_noexcept(itemFirst)))) { while (currentTailIndex != stopIndex) { new ((*this->tailBlock)[currentTailIndex++]) T(*itemFirst++); } @@ -2636,7 +2705,7 @@ class ConcurrentQueue else { MOODYCAMEL_TRY { while (currentTailIndex != stopIndex) { - new ((*this->tailBlock)[currentTailIndex]) T(details::nomove_if<(bool)!MOODYCAMEL_NOEXCEPT_CTOR(T, decltype(*itemFirst), new (nullptr) T(details::deref_noexcept(itemFirst)))>::eval(*itemFirst)); + new ((*this->tailBlock)[currentTailIndex]) T(details::nomove_if(nullptr)) T(details::deref_noexcept(itemFirst)))>::eval(*itemFirst)); ++currentTailIndex; ++itemFirst; } @@ -2688,6 +2757,9 @@ class ConcurrentQueue this->tailIndex.store(newTailIndex, std::memory_order_release); return true; } +#ifdef _MSC_VER +#pragma warning(pop) +#endif template size_t dequeue_bulk(It& itemFirst, size_t max) @@ -2700,7 +2772,6 @@ class ConcurrentQueue std::atomic_thread_fence(std::memory_order_acquire); auto myDequeueCount = this->dequeueOptimisticCount.fetch_add(desiredCount, std::memory_order_relaxed); - assert(overcommit <= myDequeueCount); tail = this->tailIndex.load(std::memory_order_acquire); auto actualCount = static_cast(tail - (myDequeueCount - overcommit)); @@ -2720,7 +2791,7 @@ class ConcurrentQueue auto indexIndex = get_block_index_index_for_index(index, localBlockIndex); do { auto blockStartIndex = index; - auto endIndex = (index & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); + index_t endIndex = (index & ~static_cast(BLOCK_SIZE - 1)) + static_cast(BLOCK_SIZE); endIndex = details::circular_less_than(firstIndex + static_cast(actualCount), endIndex) ? firstIndex + static_cast(actualCount) : endIndex; auto entry = localBlockIndex->index[indexIndex]; @@ -2752,7 +2823,7 @@ class ConcurrentQueue } if (block->ConcurrentQueue::Block::template set_many_empty(blockStartIndex, static_cast(endIndex - blockStartIndex))) { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX debug::DebugLock lock(mutex); #endif entry->value.store(nullptr, std::memory_order_relaxed); @@ -2770,7 +2841,7 @@ class ConcurrentQueue } if (block->ConcurrentQueue::Block::template set_many_empty(blockStartIndex, static_cast(endIndex - blockStartIndex))) { { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX debug::DebugLock lock(mutex); #endif // Note that the set_many_empty above did a release, meaning that anybody who acquires the block @@ -2818,7 +2889,7 @@ class ConcurrentQueue if (localBlockIndex == nullptr) { return false; // this can happen if new_block_index failed in the constructor } - auto newTail = (localBlockIndex->tail.load(std::memory_order_relaxed) + 1) & (localBlockIndex->capacity - 1); + size_t newTail = (localBlockIndex->tail.load(std::memory_order_relaxed) + 1) & (localBlockIndex->capacity - 1); idxEntry = localBlockIndex->index[newTail]; if (idxEntry->key.load(std::memory_order_relaxed) == INVALID_BLOCK_BASE || idxEntry->value.load(std::memory_order_relaxed) == nullptr) { @@ -2829,7 +2900,10 @@ class ConcurrentQueue } // No room in the old block index, try to allocate another one! - if (allocMode == CannotAlloc || !new_block_index()) { + MOODYCAMEL_CONSTEXPR_IF (allocMode == CannotAlloc) { + return false; + } + else if (!new_block_index()) { return false; } localBlockIndex = blockIndex.load(std::memory_order_relaxed); @@ -2856,7 +2930,7 @@ class ConcurrentQueue inline size_t get_block_index_index_for_index(index_t index, BlockIndexHeader*& localBlockIndex) const { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX debug::DebugLock lock(mutex); #endif index &= ~static_cast(BLOCK_SIZE - 1); @@ -2932,10 +3006,10 @@ class ConcurrentQueue private: #endif -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODBLOCKINDEX mutable debug::DebugMutex mutex; #endif -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM friend struct MemStats; #endif }; @@ -2975,7 +3049,7 @@ class ConcurrentQueue inline void add_block_to_free_list(Block* block) { -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM block->owner = nullptr; #endif freeList.add(block); @@ -3009,15 +3083,16 @@ class ConcurrentQueue return block; } - if (canAlloc == CanAlloc) { + MOODYCAMEL_CONSTEXPR_IF (canAlloc == CanAlloc) { return create(); } - - return nullptr; + else { + return nullptr; + } } -#if MCDBGQ_TRACKMEM +#ifdef MCDBGQ_TRACKMEM public: struct MemStats { size_t allocatedBlocks; @@ -3135,7 +3210,7 @@ class ConcurrentQueue ProducerBase* recycle_or_create_producer(bool isExplicit, bool& recycled) { -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH debug::DebugLock lock(implicitProdMutex); #endif // Try to re-use one first @@ -3242,50 +3317,56 @@ class ConcurrentQueue inline void populate_initial_implicit_producer_hash() { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return; - - implicitProducerHashCount.store(0, std::memory_order_relaxed); - auto hash = &initialImplicitProducerHash; - hash->capacity = INITIAL_IMPLICIT_PRODUCER_HASH_SIZE; - hash->entries = &initialImplicitProducerHashEntries[0]; - for (size_t i = 0; i != INITIAL_IMPLICIT_PRODUCER_HASH_SIZE; ++i) { - initialImplicitProducerHashEntries[i].key.store(details::invalid_thread_id, std::memory_order_relaxed); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) { + return; + } + else { + implicitProducerHashCount.store(0, std::memory_order_relaxed); + auto hash = &initialImplicitProducerHash; + hash->capacity = INITIAL_IMPLICIT_PRODUCER_HASH_SIZE; + hash->entries = &initialImplicitProducerHashEntries[0]; + for (size_t i = 0; i != INITIAL_IMPLICIT_PRODUCER_HASH_SIZE; ++i) { + initialImplicitProducerHashEntries[i].key.store(details::invalid_thread_id, std::memory_order_relaxed); + } + hash->prev = nullptr; + implicitProducerHash.store(hash, std::memory_order_relaxed); } - hash->prev = nullptr; - implicitProducerHash.store(hash, std::memory_order_relaxed); } void swap_implicit_producer_hashes(ConcurrentQueue& other) { - if (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) return; - - // Swap (assumes our implicit producer hash is initialized) - initialImplicitProducerHashEntries.swap(other.initialImplicitProducerHashEntries); - initialImplicitProducerHash.entries = &initialImplicitProducerHashEntries[0]; - other.initialImplicitProducerHash.entries = &other.initialImplicitProducerHashEntries[0]; - - details::swap_relaxed(implicitProducerHashCount, other.implicitProducerHashCount); - - details::swap_relaxed(implicitProducerHash, other.implicitProducerHash); - if (implicitProducerHash.load(std::memory_order_relaxed) == &other.initialImplicitProducerHash) { - implicitProducerHash.store(&initialImplicitProducerHash, std::memory_order_relaxed); + MOODYCAMEL_CONSTEXPR_IF (INITIAL_IMPLICIT_PRODUCER_HASH_SIZE == 0) { + return; } else { - ImplicitProducerHash* hash; - for (hash = implicitProducerHash.load(std::memory_order_relaxed); hash->prev != &other.initialImplicitProducerHash; hash = hash->prev) { - continue; + // Swap (assumes our implicit producer hash is initialized) + initialImplicitProducerHashEntries.swap(other.initialImplicitProducerHashEntries); + initialImplicitProducerHash.entries = &initialImplicitProducerHashEntries[0]; + other.initialImplicitProducerHash.entries = &other.initialImplicitProducerHashEntries[0]; + + details::swap_relaxed(implicitProducerHashCount, other.implicitProducerHashCount); + + details::swap_relaxed(implicitProducerHash, other.implicitProducerHash); + if (implicitProducerHash.load(std::memory_order_relaxed) == &other.initialImplicitProducerHash) { + implicitProducerHash.store(&initialImplicitProducerHash, std::memory_order_relaxed); } - hash->prev = &initialImplicitProducerHash; - } - if (other.implicitProducerHash.load(std::memory_order_relaxed) == &initialImplicitProducerHash) { - other.implicitProducerHash.store(&other.initialImplicitProducerHash, std::memory_order_relaxed); - } - else { - ImplicitProducerHash* hash; - for (hash = other.implicitProducerHash.load(std::memory_order_relaxed); hash->prev != &initialImplicitProducerHash; hash = hash->prev) { - continue; + else { + ImplicitProducerHash* hash; + for (hash = implicitProducerHash.load(std::memory_order_relaxed); hash->prev != &other.initialImplicitProducerHash; hash = hash->prev) { + continue; + } + hash->prev = &initialImplicitProducerHash; + } + if (other.implicitProducerHash.load(std::memory_order_relaxed) == &initialImplicitProducerHash) { + other.implicitProducerHash.store(&other.initialImplicitProducerHash, std::memory_order_relaxed); + } + else { + ImplicitProducerHash* hash; + for (hash = other.implicitProducerHash.load(std::memory_order_relaxed); hash->prev != &initialImplicitProducerHash; hash = hash->prev) { + continue; + } + hash->prev = &other.initialImplicitProducerHash; } - hash->prev = &other.initialImplicitProducerHash; } } @@ -3302,7 +3383,7 @@ class ConcurrentQueue // Code and algorithm adapted from http://preshing.com/20130605/the-worlds-simplest-lock-free-hash-table -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH debug::DebugLock lock(implicitProdMutex); #endif @@ -3310,6 +3391,7 @@ class ConcurrentQueue auto hashedId = details::hash_thread_id(id); auto mainHash = implicitProducerHash.load(std::memory_order_acquire); + assert(mainHash != nullptr); // silence clang-tidy and MSVC warnings (hash cannot be null) for (auto hash = mainHash; hash != nullptr; hash = hash->prev) { // Look for the id in this hash auto index = hashedId; @@ -3356,6 +3438,7 @@ class ConcurrentQueue // Insert! auto newCount = 1 + implicitProducerHashCount.fetch_add(1, std::memory_order_relaxed); while (true) { + // NOLINTNEXTLINE(clang-analyzer-core.NullDereference) if (newCount >= (mainHash->capacity >> 1) && !implicitProducerHashResizeInProgress.test_and_set(std::memory_order_acquire)) { // We've acquired the resize lock, try to allocate a bigger hash table. // Note the acquire fence synchronizes with the release fence at the end of this block, and hence when @@ -3376,7 +3459,7 @@ class ConcurrentQueue } auto newHash = new (raw) ImplicitProducerHash; - newHash->capacity = newCapacity; + newHash->capacity = static_cast(newCapacity); newHash->entries = reinterpret_cast(details::align_for(raw + sizeof(ImplicitProducerHash))); for (size_t i = 0; i != newCapacity; ++i) { new (newHash->entries + i) ImplicitProducerKVP; @@ -3447,7 +3530,7 @@ class ConcurrentQueue details::ThreadExitNotifier::unsubscribe(&producer->threadExitListener); // Remove from hash -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH debug::DebugLock lock(implicitProdMutex); #endif auto hash = implicitProducerHash.load(std::memory_order_acquire); @@ -3486,55 +3569,76 @@ class ConcurrentQueue ////////////////////////////////// // Utility functions ////////////////////////////////// - + + template + static inline void* aligned_malloc(size_t size) + { + MOODYCAMEL_CONSTEXPR_IF (std::alignment_of::value <= std::alignment_of::value) + return (Traits::malloc)(size); + else { + size_t alignment = std::alignment_of::value; + void* raw = (Traits::malloc)(size + alignment - 1 + sizeof(void*)); + if (!raw) + return nullptr; + char* ptr = details::align_for(reinterpret_cast(raw) + sizeof(void*)); + *(reinterpret_cast(ptr) - 1) = raw; + return ptr; + } + } + + template + static inline void aligned_free(void* ptr) + { + MOODYCAMEL_CONSTEXPR_IF (std::alignment_of::value <= std::alignment_of::value) + return (Traits::free)(ptr); + else + (Traits::free)(ptr ? *(reinterpret_cast(ptr) - 1) : nullptr); + } + template static inline U* create_array(size_t count) { assert(count > 0); - auto p = static_cast((Traits::malloc)(sizeof(U) * count)); - if (p == nullptr) { + U* p = static_cast(aligned_malloc(sizeof(U) * count)); + if (p == nullptr) return nullptr; - } - - for (size_t i = 0; i != count; ++i) { + + for (size_t i = 0; i != count; ++i) new (p + i) U(); - } return p; } - + template static inline void destroy_array(U* p, size_t count) { if (p != nullptr) { assert(count > 0); - for (size_t i = count; i != 0; ) { + for (size_t i = count; i != 0; ) (p + --i)->~U(); - } - (Traits::free)(p); } + aligned_free(p); } - + template static inline U* create() { - auto p = (Traits::malloc)(sizeof(U)); + void* p = aligned_malloc(sizeof(U)); return p != nullptr ? new (p) U : nullptr; } - + template static inline U* create(A1&& a1) { - auto p = (Traits::malloc)(sizeof(U)); + void* p = aligned_malloc(sizeof(U)); return p != nullptr ? new (p) U(std::forward(a1)) : nullptr; } - + template static inline void destroy(U* p) { - if (p != nullptr) { + if (p != nullptr) p->~U(); - } - (Traits::free)(p); + aligned_free(p); } private: @@ -3545,7 +3649,7 @@ class ConcurrentQueue Block* initialBlockPool; size_t initialBlockPoolSize; -#if !MCDBGQ_USEDEBUGFREELIST +#ifndef MCDBGQ_USEDEBUGFREELIST FreeList freeList; #else debug::DebugFreeList freeList; @@ -3560,7 +3664,7 @@ class ConcurrentQueue std::atomic nextExplicitConsumerId; std::atomic globalExplicitConsumerOffset; -#if MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH +#ifdef MCDBGQ_NOLOCKFREE_IMPLICITPRODHASH debug::DebugMutex implicitProdMutex; #endif @@ -3594,7 +3698,7 @@ ConsumerToken::ConsumerToken(ConcurrentQueue& queue) : itemsConsumedFromCurrent(0), currentProducer(nullptr), desiredProducer(nullptr) { initialOffset = queue.nextExplicitConsumerId.fetch_add(1, std::memory_order_release); - lastKnownGlobalOffset = -1; + lastKnownGlobalOffset = static_cast(-1); } template @@ -3602,7 +3706,7 @@ ConsumerToken::ConsumerToken(BlockingConcurrentQueue& queue) : itemsConsumedFromCurrent(0), currentProducer(nullptr), desiredProducer(nullptr) { initialOffset = reinterpret_cast*>(&queue)->nextExplicitConsumerId.fetch_add(1, std::memory_order_release); - lastKnownGlobalOffset = -1; + lastKnownGlobalOffset = static_cast(-1); } template @@ -3629,6 +3733,10 @@ inline void swap(typename ConcurrentQueue::ImplicitProducerKVP& a, ty } -#if defined(__GNUC__) +#if defined(_MSC_VER) && (!defined(_HAS_CXX17) || !_HAS_CXX17) +#pragma warning(pop) +#endif + +#if defined(__GNUC__) && !defined(__INTEL_COMPILER) #pragma GCC diagnostic pop #endif diff --git a/moodycamel/lightweightsemaphore.h b/moodycamel/lightweightsemaphore.h new file mode 100755 index 000000000..b0f24e1cd --- /dev/null +++ b/moodycamel/lightweightsemaphore.h @@ -0,0 +1,411 @@ +// Provides an efficient implementation of a semaphore (LightweightSemaphore). +// This is an extension of Jeff Preshing's sempahore implementation (licensed +// under the terms of its separate zlib license) that has been adapted and +// extended by Cameron Desrochers. + +#pragma once + +#include // For std::size_t +#include +#include // For std::make_signed + +#if defined(_WIN32) +// Avoid including windows.h in a header; we only need a handful of +// items, so we'll redeclare them here (this is relatively safe since +// the API generally has to remain stable between Windows versions). +// I know this is an ugly hack but it still beats polluting the global +// namespace with thousands of generic names or adding a .cpp for nothing. +extern "C" { + struct _SECURITY_ATTRIBUTES; + __declspec(dllimport) void* __stdcall CreateSemaphoreW(_SECURITY_ATTRIBUTES* lpSemaphoreAttributes, long lInitialCount, long lMaximumCount, const wchar_t* lpName); + __declspec(dllimport) int __stdcall CloseHandle(void* hObject); + __declspec(dllimport) unsigned long __stdcall WaitForSingleObject(void* hHandle, unsigned long dwMilliseconds); + __declspec(dllimport) int __stdcall ReleaseSemaphore(void* hSemaphore, long lReleaseCount, long* lpPreviousCount); +} +#elif defined(__MACH__) +#include +#elif defined(__unix__) +#include +#endif + +namespace moodycamel +{ +namespace details +{ + +// Code in the mpmc_sema namespace below is an adaptation of Jeff Preshing's +// portable + lightweight semaphore implementations, originally from +// https://github.com/preshing/cpp11-on-multicore/blob/master/common/sema.h +// LICENSE: +// Copyright (c) 2015 Jeff Preshing +// +// This software is provided 'as-is', without any express or implied +// warranty. In no event will the authors be held liable for any damages +// arising from the use of this software. +// +// Permission is granted to anyone to use this software for any purpose, +// including commercial applications, and to alter it and redistribute it +// freely, subject to the following restrictions: +// +// 1. The origin of this software must not be misrepresented; you must not +// claim that you wrote the original software. If you use this software +// in a product, an acknowledgement in the product documentation would be +// appreciated but is not required. +// 2. Altered source versions must be plainly marked as such, and must not be +// misrepresented as being the original software. +// 3. This notice may not be removed or altered from any source distribution. +#if defined(_WIN32) +class Semaphore +{ +private: + void* m_hSema; + + Semaphore(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; + Semaphore& operator=(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; + +public: + Semaphore(int initialCount = 0) + { + assert(initialCount >= 0); + const long maxLong = 0x7fffffff; + m_hSema = CreateSemaphoreW(nullptr, initialCount, maxLong, nullptr); + assert(m_hSema); + } + + ~Semaphore() + { + CloseHandle(m_hSema); + } + + bool wait() + { + const unsigned long infinite = 0xffffffff; + return WaitForSingleObject(m_hSema, infinite) == 0; + } + + bool try_wait() + { + return WaitForSingleObject(m_hSema, 0) == 0; + } + + bool timed_wait(std::uint64_t usecs) + { + return WaitForSingleObject(m_hSema, (unsigned long)(usecs / 1000)) == 0; + } + + void signal(int count = 1) + { + while (!ReleaseSemaphore(m_hSema, count, nullptr)); + } +}; +#elif defined(__MACH__) +//--------------------------------------------------------- +// Semaphore (Apple iOS and OSX) +// Can't use POSIX semaphores due to http://lists.apple.com/archives/darwin-kernel/2009/Apr/msg00010.html +//--------------------------------------------------------- +class Semaphore +{ +private: + semaphore_t m_sema; + + Semaphore(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; + Semaphore& operator=(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; + +public: + Semaphore(int initialCount = 0) + { + assert(initialCount >= 0); + kern_return_t rc = semaphore_create(mach_task_self(), &m_sema, SYNC_POLICY_FIFO, initialCount); + assert(rc == KERN_SUCCESS); + (void)rc; + } + + ~Semaphore() + { + semaphore_destroy(mach_task_self(), m_sema); + } + + bool wait() + { + return semaphore_wait(m_sema) == KERN_SUCCESS; + } + + bool try_wait() + { + return timed_wait(0); + } + + bool timed_wait(std::uint64_t timeout_usecs) + { + mach_timespec_t ts; + ts.tv_sec = static_cast(timeout_usecs / 1000000); + ts.tv_nsec = static_cast((timeout_usecs % 1000000) * 1000); + + // added in OSX 10.10: https://developer.apple.com/library/prerelease/mac/documentation/General/Reference/APIDiffsMacOSX10_10SeedDiff/modules/Darwin.html + kern_return_t rc = semaphore_timedwait(m_sema, ts); + return rc == KERN_SUCCESS; + } + + void signal() + { + while (semaphore_signal(m_sema) != KERN_SUCCESS); + } + + void signal(int count) + { + while (count-- > 0) + { + while (semaphore_signal(m_sema) != KERN_SUCCESS); + } + } +}; +#elif defined(__unix__) +//--------------------------------------------------------- +// Semaphore (POSIX, Linux) +//--------------------------------------------------------- +class Semaphore +{ +private: + sem_t m_sema; + + Semaphore(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; + Semaphore& operator=(const Semaphore& other) MOODYCAMEL_DELETE_FUNCTION; + +public: + Semaphore(int initialCount = 0) + { + assert(initialCount >= 0); + int rc = sem_init(&m_sema, 0, static_cast(initialCount)); + assert(rc == 0); + (void)rc; + } + + ~Semaphore() + { + sem_destroy(&m_sema); + } + + bool wait() + { + // http://stackoverflow.com/questions/2013181/gdb-causes-sem-wait-to-fail-with-eintr-error + int rc; + do { + rc = sem_wait(&m_sema); + } while (rc == -1 && errno == EINTR); + return rc == 0; + } + + bool try_wait() + { + int rc; + do { + rc = sem_trywait(&m_sema); + } while (rc == -1 && errno == EINTR); + return rc == 0; + } + + bool timed_wait(std::uint64_t usecs) + { + struct timespec ts; + const int usecs_in_1_sec = 1000000; + const int nsecs_in_1_sec = 1000000000; + clock_gettime(CLOCK_REALTIME, &ts); + ts.tv_sec += (time_t)(usecs / usecs_in_1_sec); + ts.tv_nsec += (long)(usecs % usecs_in_1_sec) * 1000; + // sem_timedwait bombs if you have more than 1e9 in tv_nsec + // so we have to clean things up before passing it in + if (ts.tv_nsec >= nsecs_in_1_sec) { + ts.tv_nsec -= nsecs_in_1_sec; + ++ts.tv_sec; + } + + int rc; + do { + rc = sem_timedwait(&m_sema, &ts); + } while (rc == -1 && errno == EINTR); + return rc == 0; + } + + void signal() + { + while (sem_post(&m_sema) == -1); + } + + void signal(int count) + { + while (count-- > 0) + { + while (sem_post(&m_sema) == -1); + } + } +}; +#else +#error Unsupported platform! (No semaphore wrapper available) +#endif + +} // end namespace details + + +//--------------------------------------------------------- +// LightweightSemaphore +//--------------------------------------------------------- +class LightweightSemaphore +{ +public: + typedef std::make_signed::type ssize_t; + +private: + std::atomic m_count; + details::Semaphore m_sema; + int m_maxSpins; + + bool waitWithPartialSpinning(std::int64_t timeout_usecs = -1) + { + ssize_t oldCount; + int spin = m_maxSpins; + while (--spin >= 0) + { + oldCount = m_count.load(std::memory_order_relaxed); + if ((oldCount > 0) && m_count.compare_exchange_strong(oldCount, oldCount - 1, std::memory_order_acquire, std::memory_order_relaxed)) + return true; + std::atomic_signal_fence(std::memory_order_acquire); // Prevent the compiler from collapsing the loop. + } + oldCount = m_count.fetch_sub(1, std::memory_order_acquire); + if (oldCount > 0) + return true; + if (timeout_usecs < 0) + { + if (m_sema.wait()) + return true; + } + if (timeout_usecs > 0 && m_sema.timed_wait((std::uint64_t)timeout_usecs)) + return true; + // At this point, we've timed out waiting for the semaphore, but the + // count is still decremented indicating we may still be waiting on + // it. So we have to re-adjust the count, but only if the semaphore + // wasn't signaled enough times for us too since then. If it was, we + // need to release the semaphore too. + while (true) + { + oldCount = m_count.load(std::memory_order_acquire); + if (oldCount >= 0 && m_sema.try_wait()) + return true; + if (oldCount < 0 && m_count.compare_exchange_strong(oldCount, oldCount + 1, std::memory_order_relaxed, std::memory_order_relaxed)) + return false; + } + } + + ssize_t waitManyWithPartialSpinning(ssize_t max, std::int64_t timeout_usecs = -1) + { + assert(max > 0); + ssize_t oldCount; + int spin = m_maxSpins; + while (--spin >= 0) + { + oldCount = m_count.load(std::memory_order_relaxed); + if (oldCount > 0) + { + ssize_t newCount = oldCount > max ? oldCount - max : 0; + if (m_count.compare_exchange_strong(oldCount, newCount, std::memory_order_acquire, std::memory_order_relaxed)) + return oldCount - newCount; + } + std::atomic_signal_fence(std::memory_order_acquire); + } + oldCount = m_count.fetch_sub(1, std::memory_order_acquire); + if (oldCount <= 0) + { + if ((timeout_usecs == 0) || (timeout_usecs < 0 && !m_sema.wait()) || (timeout_usecs > 0 && !m_sema.timed_wait((std::uint64_t)timeout_usecs))) + { + while (true) + { + oldCount = m_count.load(std::memory_order_acquire); + if (oldCount >= 0 && m_sema.try_wait()) + break; + if (oldCount < 0 && m_count.compare_exchange_strong(oldCount, oldCount + 1, std::memory_order_relaxed, std::memory_order_relaxed)) + return 0; + } + } + } + if (max > 1) + return 1 + tryWaitMany(max - 1); + return 1; + } + +public: + LightweightSemaphore(ssize_t initialCount = 0, int maxSpins = 10000) : m_count(initialCount), m_maxSpins(maxSpins) + { + assert(initialCount >= 0); + assert(maxSpins >= 0); + } + + bool tryWait() + { + ssize_t oldCount = m_count.load(std::memory_order_relaxed); + while (oldCount > 0) + { + if (m_count.compare_exchange_weak(oldCount, oldCount - 1, std::memory_order_acquire, std::memory_order_relaxed)) + return true; + } + return false; + } + + bool wait() + { + return tryWait() || waitWithPartialSpinning(); + } + + bool wait(std::int64_t timeout_usecs) + { + return tryWait() || waitWithPartialSpinning(timeout_usecs); + } + + // Acquires between 0 and (greedily) max, inclusive + ssize_t tryWaitMany(ssize_t max) + { + assert(max >= 0); + ssize_t oldCount = m_count.load(std::memory_order_relaxed); + while (oldCount > 0) + { + ssize_t newCount = oldCount > max ? oldCount - max : 0; + if (m_count.compare_exchange_weak(oldCount, newCount, std::memory_order_acquire, std::memory_order_relaxed)) + return oldCount - newCount; + } + return 0; + } + + // Acquires at least one, and (greedily) at most max + ssize_t waitMany(ssize_t max, std::int64_t timeout_usecs) + { + assert(max >= 0); + ssize_t result = tryWaitMany(max); + if (result == 0 && max > 0) + result = waitManyWithPartialSpinning(max, timeout_usecs); + return result; + } + + ssize_t waitMany(ssize_t max) + { + ssize_t result = waitMany(max, -1); + assert(result > 0); + return result; + } + + void signal(ssize_t count = 1) + { + assert(count >= 0); + ssize_t oldCount = m_count.fetch_add(count, std::memory_order_release); + ssize_t toRelease = -oldCount < count ? -oldCount : count; + if (toRelease > 0) + { + m_sema.signal((int)toRelease); + } + } + + std::size_t availableApprox() const + { + ssize_t count = m_count.load(std::memory_order_relaxed); + return count > 0 ? static_cast(count) : 0; + } +}; + +} // end namespace moodycamel diff --git a/moodycamel/readerwritercircularbuffer.h b/moodycamel/readerwritercircularbuffer.h new file mode 100755 index 000000000..ea25df59d --- /dev/null +++ b/moodycamel/readerwritercircularbuffer.h @@ -0,0 +1,288 @@ +// ©2020 Cameron Desrochers. +// Distributed under the simplified BSD license (see the license file that +// should have come with this header). + +// Provides a C++11 implementation of a single-producer, single-consumer wait-free concurrent +// circular buffer (fixed-size queue). + +#pragma once + +#include +#include +#include +#include +#include +#include + +// Note that this implementation is fully modern C++11 (not compatible with old MSVC versions) +// but we still include atomicops.h for its LightweightSemaphore implementation. +#include "atomicops.h" + +#ifndef MOODYCAMEL_CACHE_LINE_SIZE +#define MOODYCAMEL_CACHE_LINE_SIZE 64 +#endif + +namespace moodycamel { + +template +class BlockingReaderWriterCircularBuffer +{ +public: + typedef T value_type; + +public: + explicit BlockingReaderWriterCircularBuffer(std::size_t capacity) + : maxcap(capacity), mask(), rawData(), data(), + slots_(new spsc_sema::LightweightSemaphore(static_cast(capacity))), + items(new spsc_sema::LightweightSemaphore(0)), + nextSlot(0), nextItem(0) + { + // Round capacity up to power of two to compute modulo mask. + // Adapted from http://graphics.stanford.edu/~seander/bithacks.html#RoundUpPowerOf2 + --capacity; + capacity |= capacity >> 1; + capacity |= capacity >> 2; + capacity |= capacity >> 4; + for (std::size_t i = 1; i < sizeof(std::size_t); i <<= 1) + capacity |= capacity >> (i << 3); + mask = capacity++; + rawData = static_cast(std::malloc(capacity * sizeof(T) + std::alignment_of::value - 1)); + data = align_for(rawData); + } + + BlockingReaderWriterCircularBuffer(BlockingReaderWriterCircularBuffer&& other) + : maxcap(0), mask(0), rawData(nullptr), data(nullptr), + slots_(new spsc_sema::LightweightSemaphore(0)), + items(new spsc_sema::LightweightSemaphore(0)), + nextSlot(), nextItem() + { + swap(other); + } + + BlockingReaderWriterCircularBuffer(BlockingReaderWriterCircularBuffer const&) = delete; + + // Note: The queue should not be accessed concurrently while it's + // being deleted. It's up to the user to synchronize this. + ~BlockingReaderWriterCircularBuffer() + { + for (std::size_t i = 0, n = items->availableApprox(); i != n; ++i) + reinterpret_cast(data)[(nextItem + i) & mask].~T(); + std::free(rawData); + } + + BlockingReaderWriterCircularBuffer& operator=(BlockingReaderWriterCircularBuffer&& other) noexcept + { + swap(other); + return *this; + } + + BlockingReaderWriterCircularBuffer& operator=(BlockingReaderWriterCircularBuffer const&) = delete; + + // Swaps the contents of this buffer with the contents of another. + // Not thread-safe. + void swap(BlockingReaderWriterCircularBuffer& other) noexcept + { + std::swap(maxcap, other.maxcap); + std::swap(mask, other.mask); + std::swap(rawData, other.rawData); + std::swap(data, other.data); + std::swap(slots_, other.slots_); + std::swap(items, other.items); + std::swap(nextSlot, other.nextSlot); + std::swap(nextItem, other.nextItem); + } + + // Enqueues a single item (by copying it). + // Fails if not enough room to enqueue. + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + bool try_enqueue(T const& item) + { + if (!slots_->tryWait()) + return false; + inner_enqueue(item); + return true; + } + + // Enqueues a single item (by moving it, if possible). + // Fails if not enough room to enqueue. + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + bool try_enqueue(T&& item) + { + if (!slots_->tryWait()) + return false; + inner_enqueue(std::move(item)); + return true; + } + + // Blocks the current thread until there's enough space to enqueue the given item, + // then enqueues it (via copy). + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + void wait_enqueue(T const& item) + { + while (!slots_->wait()); + inner_enqueue(item); + } + + // Blocks the current thread until there's enough space to enqueue the given item, + // then enqueues it (via move, if possible). + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + void wait_enqueue(T&& item) + { + while (!slots_->wait()); + inner_enqueue(std::move(item)); + } + + // Blocks the current thread until there's enough space to enqueue the given item, + // or the timeout expires. Returns false without enqueueing the item if the timeout + // expires, otherwise enqueues the item (via copy) and returns true. + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + bool wait_enqueue_timed(T const& item, std::int64_t timeout_usecs) + { + if (!slots_->wait(timeout_usecs)) + return false; + inner_enqueue(item); + return true; + } + + // Blocks the current thread until there's enough space to enqueue the given item, + // or the timeout expires. Returns false without enqueueing the item if the timeout + // expires, otherwise enqueues the item (via move, if possible) and returns true. + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + bool wait_enqueue_timed(T&& item, std::int64_t timeout_usecs) + { + if (!slots_->wait(timeout_usecs)) + return false; + inner_enqueue(std::move(item)); + return true; + } + + // Blocks the current thread until there's enough space to enqueue the given item, + // or the timeout expires. Returns false without enqueueing the item if the timeout + // expires, otherwise enqueues the item (via copy) and returns true. + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + template + inline bool wait_enqueue_timed(T const& item, std::chrono::duration const& timeout) + { + return wait_enqueue_timed(item, std::chrono::duration_cast(timeout).count()); + } + + // Blocks the current thread until there's enough space to enqueue the given item, + // or the timeout expires. Returns false without enqueueing the item if the timeout + // expires, otherwise enqueues the item (via move, if possible) and returns true. + // Thread-safe when called by producer thread. + // No exception guarantee (state will be corrupted) if constructor of T throws. + template + inline bool wait_enqueue_timed(T&& item, std::chrono::duration const& timeout) + { + return wait_enqueue_timed(std::move(item), std::chrono::duration_cast(timeout).count()); + } + + // Attempts to dequeue a single item. + // Returns false if the buffer is empty. + // Thread-safe when called by consumer thread. + // No exception guarantee (state will be corrupted) if assignment operator of U throws. + template + bool try_dequeue(U& item) + { + if (!items->tryWait()) + return false; + inner_dequeue(item); + return true; + } + + // Blocks the current thread until there's something to dequeue, then dequeues it. + // Thread-safe when called by consumer thread. + // No exception guarantee (state will be corrupted) if assignment operator of U throws. + template + void wait_dequeue(U& item) + { + while (!items->wait()); + inner_dequeue(item); + } + + // Blocks the current thread until either there's something to dequeue + // or the timeout expires. Returns false without setting `item` if the + // timeout expires, otherwise assigns to `item` and returns true. + // Thread-safe when called by consumer thread. + // No exception guarantee (state will be corrupted) if assignment operator of U throws. + template + bool wait_dequeue_timed(U& item, std::int64_t timeout_usecs) + { + if (!items->wait(timeout_usecs)) + return false; + inner_dequeue(item); + return true; + } + + // Blocks the current thread until either there's something to dequeue + // or the timeout expires. Returns false without setting `item` if the + // timeout expires, otherwise assigns to `item` and returns true. + // Thread-safe when called by consumer thread. + // No exception guarantee (state will be corrupted) if assignment operator of U throws. + template + inline bool wait_dequeue_timed(U& item, std::chrono::duration const& timeout) + { + return wait_dequeue_timed(item, std::chrono::duration_cast(timeout).count()); + } + + // Returns a (possibly outdated) snapshot of the total number of elements currently in the buffer. + // Thread-safe. + inline std::size_t size_approx() const + { + return items->availableApprox(); + } + + // Returns the maximum number of elements that this circular buffer can hold at once. + // Thread-safe. + inline std::size_t max_capacity() const + { + return maxcap; + } + +private: + template + void inner_enqueue(U&& item) + { + std::size_t i = nextSlot++; + new (reinterpret_cast(data) + (i & mask)) T(std::forward(item)); + items->signal(); + } + + template + void inner_dequeue(U& item) + { + std::size_t i = nextItem++; + T& element = reinterpret_cast(data)[i & mask]; + item = std::move(element); + element.~T(); + slots_->signal(); + } + + template + static inline char* align_for(char* ptr) + { + const std::size_t alignment = std::alignment_of::value; + return ptr + (alignment - (reinterpret_cast(ptr) % alignment)) % alignment; + } + +private: + std::size_t maxcap; // actual (non-power-of-two) capacity + std::size_t mask; // circular buffer capacity mask (for cheap modulo) + char* rawData; // raw circular buffer memory + char* data; // circular buffer memory aligned to element alignment + std::unique_ptr slots_; // number of slots currently free (named with underscore to accommodate Qt's 'slots' macro) + std::unique_ptr items; // number of elements currently enqueued + char cachelineFiller0[MOODYCAMEL_CACHE_LINE_SIZE - sizeof(char*) * 2 - sizeof(std::size_t) * 2 - sizeof(std::unique_ptr) * 2]; + std::size_t nextSlot; // index of next free slot to enqueue into + char cachelineFiller1[MOODYCAMEL_CACHE_LINE_SIZE - sizeof(std::size_t)]; + std::size_t nextItem; // index of next element to dequeue from +}; + +} diff --git a/moodycamel/readerwriterqueue.h b/moodycamel/readerwriterqueue.h new file mode 100755 index 000000000..d87110a32 --- /dev/null +++ b/moodycamel/readerwriterqueue.h @@ -0,0 +1,979 @@ +// ©2013-2020 Cameron Desrochers. +// Distributed under the simplified BSD license (see the license file that +// should have come with this header). + +#pragma once + +#include "atomicops.h" +#include +#include +#include +#include +#include +#include +#include +#include // For malloc/free/abort & size_t +#include +#if __cplusplus > 199711L || _MSC_VER >= 1700 // C++11 or VS2012 +#include +#endif + + +// A lock-free queue for a single-consumer, single-producer architecture. +// The queue is also wait-free in the common path (except if more memory +// needs to be allocated, in which case malloc is called). +// Allocates memory sparingly, and only once if the original maximum size +// estimate is never exceeded. +// Tested on x86/x64 processors, but semantics should be correct for all +// architectures (given the right implementations in atomicops.h), provided +// that aligned integer and pointer accesses are naturally atomic. +// Note that there should only be one consumer thread and producer thread; +// Switching roles of the threads, or using multiple consecutive threads for +// one role, is not safe unless properly synchronized. +// Using the queue exclusively from one thread is fine, though a bit silly. + +#ifndef MOODYCAMEL_CACHE_LINE_SIZE +#define MOODYCAMEL_CACHE_LINE_SIZE 64 +#endif + +#ifndef MOODYCAMEL_EXCEPTIONS_ENABLED +#if (defined(_MSC_VER) && defined(_CPPUNWIND)) || (defined(__GNUC__) && defined(__EXCEPTIONS)) || (!defined(_MSC_VER) && !defined(__GNUC__)) +#define MOODYCAMEL_EXCEPTIONS_ENABLED +#endif +#endif + +#ifndef MOODYCAMEL_HAS_EMPLACE +#if !defined(_MSC_VER) || _MSC_VER >= 1800 // variadic templates: either a non-MS compiler or VS >= 2013 +#define MOODYCAMEL_HAS_EMPLACE 1 +#endif +#endif + +#ifndef MOODYCAMEL_MAYBE_ALIGN_TO_CACHELINE +#if defined (__APPLE__) && defined (__MACH__) && __cplusplus >= 201703L +// This is required to find out what deployment target we are using +#include +#if !defined(MAC_OS_X_VERSION_MIN_REQUIRED) || MAC_OS_X_VERSION_MIN_REQUIRED < MAC_OS_X_VERSION_10_14 +// C++17 new(size_t, align_val_t) is not backwards-compatible with older versions of macOS, so we can't support over-alignment in this case +#define MOODYCAMEL_MAYBE_ALIGN_TO_CACHELINE +#endif +#endif +#endif + +#ifndef MOODYCAMEL_MAYBE_ALIGN_TO_CACHELINE +#define MOODYCAMEL_MAYBE_ALIGN_TO_CACHELINE AE_ALIGN(MOODYCAMEL_CACHE_LINE_SIZE) +#endif + +#ifdef AE_VCPP +#pragma warning(push) +#pragma warning(disable: 4324) // structure was padded due to __declspec(align()) +#pragma warning(disable: 4820) // padding was added +#pragma warning(disable: 4127) // conditional expression is constant +#endif + +namespace moodycamel { + +template +class MOODYCAMEL_MAYBE_ALIGN_TO_CACHELINE ReaderWriterQueue +{ + // Design: Based on a queue-of-queues. The low-level queues are just + // circular buffers with front and tail indices indicating where the + // next element to dequeue is and where the next element can be enqueued, + // respectively. Each low-level queue is called a "block". Each block + // wastes exactly one element's worth of space to keep the design simple + // (if front == tail then the queue is empty, and can't be full). + // The high-level queue is a circular linked list of blocks; again there + // is a front and tail, but this time they are pointers to the blocks. + // The front block is where the next element to be dequeued is, provided + // the block is not empty. The back block is where elements are to be + // enqueued, provided the block is not full. + // The producer thread owns all the tail indices/pointers. The consumer + // thread owns all the front indices/pointers. Both threads read each + // other's variables, but only the owning thread updates them. E.g. After + // the consumer reads the producer's tail, the tail may change before the + // consumer is done dequeuing an object, but the consumer knows the tail + // will never go backwards, only forwards. + // If there is no room to enqueue an object, an additional block (of + // equal size to the last block) is added. Blocks are never removed. + +public: + typedef T value_type; + + // Constructs a queue that can hold at least `size` elements without further + // allocations. If more than MAX_BLOCK_SIZE elements are requested, + // then several blocks of MAX_BLOCK_SIZE each are reserved (including + // at least one extra buffer block). + AE_NO_TSAN explicit ReaderWriterQueue(size_t size = 15) +#ifndef NDEBUG + : enqueuing(false) + ,dequeuing(false) +#endif + { + assert(MAX_BLOCK_SIZE == ceilToPow2(MAX_BLOCK_SIZE) && "MAX_BLOCK_SIZE must be a power of 2"); + assert(MAX_BLOCK_SIZE >= 2 && "MAX_BLOCK_SIZE must be at least 2"); + + Block* firstBlock = nullptr; + + largestBlockSize = ceilToPow2(size + 1); // We need a spare slot to fit size elements in the block + if (largestBlockSize > MAX_BLOCK_SIZE * 2) { + // We need a spare block in case the producer is writing to a different block the consumer is reading from, and + // wants to enqueue the maximum number of elements. We also need a spare element in each block to avoid the ambiguity + // between front == tail meaning "empty" and "full". + // So the effective number of slots that are guaranteed to be usable at any time is the block size - 1 times the + // number of blocks - 1. Solving for size and applying a ceiling to the division gives us (after simplifying): + size_t initialBlockCount = (size + MAX_BLOCK_SIZE * 2 - 3) / (MAX_BLOCK_SIZE - 1); + largestBlockSize = MAX_BLOCK_SIZE; + Block* lastBlock = nullptr; + for (size_t i = 0; i != initialBlockCount; ++i) { + auto block = make_block(largestBlockSize); + if (block == nullptr) { +#ifdef MOODYCAMEL_EXCEPTIONS_ENABLED + throw std::bad_alloc(); +#else + abort(); +#endif + } + if (firstBlock == nullptr) { + firstBlock = block; + } + else { + lastBlock->next = block; + } + lastBlock = block; + block->next = firstBlock; + } + } + else { + firstBlock = make_block(largestBlockSize); + if (firstBlock == nullptr) { +#ifdef MOODYCAMEL_EXCEPTIONS_ENABLED + throw std::bad_alloc(); +#else + abort(); +#endif + } + firstBlock->next = firstBlock; + } + frontBlock = firstBlock; + tailBlock = firstBlock; + + // Make sure the reader/writer threads will have the initialized memory setup above: + fence(memory_order_sync); + } + + // Note: The queue should not be accessed concurrently while it's + // being moved. It's up to the user to synchronize this. + AE_NO_TSAN ReaderWriterQueue(ReaderWriterQueue&& other) + : frontBlock(other.frontBlock.load()), + tailBlock(other.tailBlock.load()), + largestBlockSize(other.largestBlockSize) +#ifndef NDEBUG + ,enqueuing(false) + ,dequeuing(false) +#endif + { + other.largestBlockSize = 32; + Block* b = other.make_block(other.largestBlockSize); + if (b == nullptr) { +#ifdef MOODYCAMEL_EXCEPTIONS_ENABLED + throw std::bad_alloc(); +#else + abort(); +#endif + } + b->next = b; + other.frontBlock = b; + other.tailBlock = b; + } + + // Note: The queue should not be accessed concurrently while it's + // being moved. It's up to the user to synchronize this. + ReaderWriterQueue& operator=(ReaderWriterQueue&& other) AE_NO_TSAN + { + Block* b = frontBlock.load(); + frontBlock = other.frontBlock.load(); + other.frontBlock = b; + b = tailBlock.load(); + tailBlock = other.tailBlock.load(); + other.tailBlock = b; + std::swap(largestBlockSize, other.largestBlockSize); + return *this; + } + + // Note: The queue should not be accessed concurrently while it's + // being deleted. It's up to the user to synchronize this. + AE_NO_TSAN ~ReaderWriterQueue() + { + // Make sure we get the latest version of all variables from other CPUs: + fence(memory_order_sync); + + // Destroy any remaining objects in queue and free memory + Block* frontBlock_ = frontBlock; + Block* block = frontBlock_; + do { + Block* nextBlock = block->next; + size_t blockFront = block->front; + size_t blockTail = block->tail; + + for (size_t i = blockFront; i != blockTail; i = (i + 1) & block->sizeMask) { + auto element = reinterpret_cast(block->data + i * sizeof(T)); + element->~T(); + (void)element; + } + + auto rawBlock = block->rawThis; + block->~Block(); + std::free(rawBlock); + block = nextBlock; + } while (block != frontBlock_); + } + + + // Enqueues a copy of element if there is room in the queue. + // Returns true if the element was enqueued, false otherwise. + // Does not allocate memory. + AE_FORCEINLINE bool try_enqueue(T const& element) AE_NO_TSAN + { + return inner_enqueue(element); + } + + // Enqueues a moved copy of element if there is room in the queue. + // Returns true if the element was enqueued, false otherwise. + // Does not allocate memory. + AE_FORCEINLINE bool try_enqueue(T&& element) AE_NO_TSAN + { + return inner_enqueue(std::forward(element)); + } + +#if MOODYCAMEL_HAS_EMPLACE + // Like try_enqueue() but with emplace semantics (i.e. construct-in-place). + template + AE_FORCEINLINE bool try_emplace(Args&&... args) AE_NO_TSAN + { + return inner_enqueue(std::forward(args)...); + } +#endif + + // Enqueues a copy of element on the queue. + // Allocates an additional block of memory if needed. + // Only fails (returns false) if memory allocation fails. + AE_FORCEINLINE bool enqueue(T const& element) AE_NO_TSAN + { + return inner_enqueue(element); + } + + // Enqueues a moved copy of element on the queue. + // Allocates an additional block of memory if needed. + // Only fails (returns false) if memory allocation fails. + AE_FORCEINLINE bool enqueue(T&& element) AE_NO_TSAN + { + return inner_enqueue(std::forward(element)); + } + +#if MOODYCAMEL_HAS_EMPLACE + // Like enqueue() but with emplace semantics (i.e. construct-in-place). + template + AE_FORCEINLINE bool emplace(Args&&... args) AE_NO_TSAN + { + return inner_enqueue(std::forward(args)...); + } +#endif + + // Attempts to dequeue an element; if the queue is empty, + // returns false instead. If the queue has at least one element, + // moves front to result using operator=, then returns true. + template + bool try_dequeue(U& result) AE_NO_TSAN + { +#ifndef NDEBUG + ReentrantGuard guard(this->dequeuing); +#endif + + // High-level pseudocode: + // Remember where the tail block is + // If the front block has an element in it, dequeue it + // Else + // If front block was the tail block when we entered the function, return false + // Else advance to next block and dequeue the item there + + // Note that we have to use the value of the tail block from before we check if the front + // block is full or not, in case the front block is empty and then, before we check if the + // tail block is at the front block or not, the producer fills up the front block *and + // moves on*, which would make us skip a filled block. Seems unlikely, but was consistently + // reproducible in practice. + // In order to avoid overhead in the common case, though, we do a double-checked pattern + // where we have the fast path if the front block is not empty, then read the tail block, + // then re-read the front block and check if it's not empty again, then check if the tail + // block has advanced. + + Block* frontBlock_ = frontBlock.load(); + size_t blockTail = frontBlock_->localTail; + size_t blockFront = frontBlock_->front.load(); + + if (blockFront != blockTail || blockFront != (frontBlock_->localTail = frontBlock_->tail.load())) { + fence(memory_order_acquire); + + non_empty_front_block: + // Front block not empty, dequeue from here + auto element = reinterpret_cast(frontBlock_->data + blockFront * sizeof(T)); + result = std::move(*element); + element->~T(); + + blockFront = (blockFront + 1) & frontBlock_->sizeMask; + + fence(memory_order_release); + frontBlock_->front = blockFront; + } + else if (frontBlock_ != tailBlock.load()) { + fence(memory_order_acquire); + + frontBlock_ = frontBlock.load(); + blockTail = frontBlock_->localTail = frontBlock_->tail.load(); + blockFront = frontBlock_->front.load(); + fence(memory_order_acquire); + + if (blockFront != blockTail) { + // Oh look, the front block isn't empty after all + goto non_empty_front_block; + } + + // Front block is empty but there's another block ahead, advance to it + Block* nextBlock = frontBlock_->next; + // Don't need an acquire fence here since next can only ever be set on the tailBlock, + // and we're not the tailBlock, and we did an acquire earlier after reading tailBlock which + // ensures next is up-to-date on this CPU in case we recently were at tailBlock. + + size_t nextBlockFront = nextBlock->front.load(); + size_t nextBlockTail = nextBlock->localTail = nextBlock->tail.load(); + fence(memory_order_acquire); + + // Since the tailBlock is only ever advanced after being written to, + // we know there's for sure an element to dequeue on it + assert(nextBlockFront != nextBlockTail); + AE_UNUSED(nextBlockTail); + + // We're done with this block, let the producer use it if it needs + fence(memory_order_release); // Expose possibly pending changes to frontBlock->front from last dequeue + frontBlock = frontBlock_ = nextBlock; + + compiler_fence(memory_order_release); // Not strictly needed + + auto element = reinterpret_cast(frontBlock_->data + nextBlockFront * sizeof(T)); + + result = std::move(*element); + element->~T(); + + nextBlockFront = (nextBlockFront + 1) & frontBlock_->sizeMask; + + fence(memory_order_release); + frontBlock_->front = nextBlockFront; + } + else { + // No elements in current block and no other block to advance to + return false; + } + + return true; + } + + + // Returns a pointer to the front element in the queue (the one that + // would be removed next by a call to `try_dequeue` or `pop`). If the + // queue appears empty at the time the method is called, nullptr is + // returned instead. + // Must be called only from the consumer thread. + T* peek() const AE_NO_TSAN + { +#ifndef NDEBUG + ReentrantGuard guard(this->dequeuing); +#endif + // See try_dequeue() for reasoning + + Block* frontBlock_ = frontBlock.load(); + size_t blockTail = frontBlock_->localTail; + size_t blockFront = frontBlock_->front.load(); + + if (blockFront != blockTail || blockFront != (frontBlock_->localTail = frontBlock_->tail.load())) { + fence(memory_order_acquire); + non_empty_front_block: + return reinterpret_cast(frontBlock_->data + blockFront * sizeof(T)); + } + else if (frontBlock_ != tailBlock.load()) { + fence(memory_order_acquire); + frontBlock_ = frontBlock.load(); + blockTail = frontBlock_->localTail = frontBlock_->tail.load(); + blockFront = frontBlock_->front.load(); + fence(memory_order_acquire); + + if (blockFront != blockTail) { + goto non_empty_front_block; + } + + Block* nextBlock = frontBlock_->next; + + size_t nextBlockFront = nextBlock->front.load(); + fence(memory_order_acquire); + + assert(nextBlockFront != nextBlock->tail.load()); + return reinterpret_cast(nextBlock->data + nextBlockFront * sizeof(T)); + } + + return nullptr; + } + + // Removes the front element from the queue, if any, without returning it. + // Returns true on success, or false if the queue appeared empty at the time + // `pop` was called. + bool pop() AE_NO_TSAN + { +#ifndef NDEBUG + ReentrantGuard guard(this->dequeuing); +#endif + // See try_dequeue() for reasoning + + Block* frontBlock_ = frontBlock.load(); + size_t blockTail = frontBlock_->localTail; + size_t blockFront = frontBlock_->front.load(); + + if (blockFront != blockTail || blockFront != (frontBlock_->localTail = frontBlock_->tail.load())) { + fence(memory_order_acquire); + + non_empty_front_block: + auto element = reinterpret_cast(frontBlock_->data + blockFront * sizeof(T)); + element->~T(); + + blockFront = (blockFront + 1) & frontBlock_->sizeMask; + + fence(memory_order_release); + frontBlock_->front = blockFront; + } + else if (frontBlock_ != tailBlock.load()) { + fence(memory_order_acquire); + frontBlock_ = frontBlock.load(); + blockTail = frontBlock_->localTail = frontBlock_->tail.load(); + blockFront = frontBlock_->front.load(); + fence(memory_order_acquire); + + if (blockFront != blockTail) { + goto non_empty_front_block; + } + + // Front block is empty but there's another block ahead, advance to it + Block* nextBlock = frontBlock_->next; + + size_t nextBlockFront = nextBlock->front.load(); + size_t nextBlockTail = nextBlock->localTail = nextBlock->tail.load(); + fence(memory_order_acquire); + + assert(nextBlockFront != nextBlockTail); + AE_UNUSED(nextBlockTail); + + fence(memory_order_release); + frontBlock = frontBlock_ = nextBlock; + + compiler_fence(memory_order_release); + + auto element = reinterpret_cast(frontBlock_->data + nextBlockFront * sizeof(T)); + element->~T(); + + nextBlockFront = (nextBlockFront + 1) & frontBlock_->sizeMask; + + fence(memory_order_release); + frontBlock_->front = nextBlockFront; + } + else { + // No elements in current block and no other block to advance to + return false; + } + + return true; + } + + // Returns the approximate number of items currently in the queue. + // Safe to call from both the producer and consumer threads. + inline size_t size_approx() const AE_NO_TSAN + { + size_t result = 0; + Block* frontBlock_ = frontBlock.load(); + Block* block = frontBlock_; + do { + fence(memory_order_acquire); + size_t blockFront = block->front.load(); + size_t blockTail = block->tail.load(); + result += (blockTail - blockFront) & block->sizeMask; + block = block->next.load(); + } while (block != frontBlock_); + return result; + } + + // Returns the total number of items that could be enqueued without incurring + // an allocation when this queue is empty. + // Safe to call from both the producer and consumer threads. + // + // NOTE: The actual capacity during usage may be different depending on the consumer. + // If the consumer is removing elements concurrently, the producer cannot add to + // the block the consumer is removing from until it's completely empty, except in + // the case where the producer was writing to the same block the consumer was + // reading from the whole time. + inline size_t max_capacity() const { + size_t result = 0; + Block* frontBlock_ = frontBlock.load(); + Block* block = frontBlock_; + do { + fence(memory_order_acquire); + result += block->sizeMask; + block = block->next.load(); + } while (block != frontBlock_); + return result; + } + + +private: + enum AllocationMode { CanAlloc, CannotAlloc }; + +#if MOODYCAMEL_HAS_EMPLACE + template + bool inner_enqueue(Args&&... args) AE_NO_TSAN +#else + template + bool inner_enqueue(U&& element) AE_NO_TSAN +#endif + { +#ifndef NDEBUG + ReentrantGuard guard(this->enqueuing); +#endif + + // High-level pseudocode (assuming we're allowed to alloc a new block): + // If room in tail block, add to tail + // Else check next block + // If next block is not the head block, enqueue on next block + // Else create a new block and enqueue there + // Advance tail to the block we just enqueued to + + Block* tailBlock_ = tailBlock.load(); + size_t blockFront = tailBlock_->localFront; + size_t blockTail = tailBlock_->tail.load(); + + size_t nextBlockTail = (blockTail + 1) & tailBlock_->sizeMask; + if (nextBlockTail != blockFront || nextBlockTail != (tailBlock_->localFront = tailBlock_->front.load())) { + fence(memory_order_acquire); + // This block has room for at least one more element + char* location = tailBlock_->data + blockTail * sizeof(T); +#if MOODYCAMEL_HAS_EMPLACE + new (location) T(std::forward(args)...); +#else + new (location) T(std::forward(element)); +#endif + + fence(memory_order_release); + tailBlock_->tail = nextBlockTail; + } + else { + fence(memory_order_acquire); + if (tailBlock_->next.load() != frontBlock) { + // Note that the reason we can't advance to the frontBlock and start adding new entries there + // is because if we did, then dequeue would stay in that block, eventually reading the new values, + // instead of advancing to the next full block (whose values were enqueued first and so should be + // consumed first). + + fence(memory_order_acquire); // Ensure we get latest writes if we got the latest frontBlock + + // tailBlock is full, but there's a free block ahead, use it + Block* tailBlockNext = tailBlock_->next.load(); + size_t nextBlockFront = tailBlockNext->localFront = tailBlockNext->front.load(); + nextBlockTail = tailBlockNext->tail.load(); + fence(memory_order_acquire); + + // This block must be empty since it's not the head block and we + // go through the blocks in a circle + assert(nextBlockFront == nextBlockTail); + tailBlockNext->localFront = nextBlockFront; + + char* location = tailBlockNext->data + nextBlockTail * sizeof(T); +#if MOODYCAMEL_HAS_EMPLACE + new (location) T(std::forward(args)...); +#else + new (location) T(std::forward(element)); +#endif + + tailBlockNext->tail = (nextBlockTail + 1) & tailBlockNext->sizeMask; + + fence(memory_order_release); + tailBlock = tailBlockNext; + } + else if (canAlloc == CanAlloc) { + // tailBlock is full and there's no free block ahead; create a new block + auto newBlockSize = largestBlockSize >= MAX_BLOCK_SIZE ? largestBlockSize : largestBlockSize * 2; + auto newBlock = make_block(newBlockSize); + if (newBlock == nullptr) { + // Could not allocate a block! + return false; + } + largestBlockSize = newBlockSize; + +#if MOODYCAMEL_HAS_EMPLACE + new (newBlock->data) T(std::forward(args)...); +#else + new (newBlock->data) T(std::forward(element)); +#endif + assert(newBlock->front == 0); + newBlock->tail = newBlock->localTail = 1; + + newBlock->next = tailBlock_->next.load(); + tailBlock_->next = newBlock; + + // Might be possible for the dequeue thread to see the new tailBlock->next + // *without* seeing the new tailBlock value, but this is OK since it can't + // advance to the next block until tailBlock is set anyway (because the only + // case where it could try to read the next is if it's already at the tailBlock, + // and it won't advance past tailBlock in any circumstance). + + fence(memory_order_release); + tailBlock = newBlock; + } + else if (canAlloc == CannotAlloc) { + // Would have had to allocate a new block to enqueue, but not allowed + return false; + } + else { + assert(false && "Should be unreachable code"); + return false; + } + } + + return true; + } + + + // Disable copying + ReaderWriterQueue(ReaderWriterQueue const&) { } + + // Disable assignment + ReaderWriterQueue& operator=(ReaderWriterQueue const&) { } + + + AE_FORCEINLINE static size_t ceilToPow2(size_t x) + { + // From http://graphics.stanford.edu/~seander/bithacks.html#RoundUpPowerOf2 + --x; + x |= x >> 1; + x |= x >> 2; + x |= x >> 4; + for (size_t i = 1; i < sizeof(size_t); i <<= 1) { + x |= x >> (i << 3); + } + ++x; + return x; + } + + template + static AE_FORCEINLINE char* align_for(char* ptr) AE_NO_TSAN + { + const std::size_t alignment = std::alignment_of::value; + return ptr + (alignment - (reinterpret_cast(ptr) % alignment)) % alignment; + } +private: +#ifndef NDEBUG + struct ReentrantGuard + { + AE_NO_TSAN ReentrantGuard(weak_atomic& _inSection) + : inSection(_inSection) + { + assert(!inSection && "Concurrent (or re-entrant) enqueue or dequeue operation detected (only one thread at a time may hold the producer or consumer role)"); + inSection = true; + } + + AE_NO_TSAN ~ReentrantGuard() { inSection = false; } + + private: + ReentrantGuard& operator=(ReentrantGuard const&); + + private: + weak_atomic& inSection; + }; +#endif + + struct Block + { + // Avoid false-sharing by putting highly contended variables on their own cache lines + weak_atomic front; // (Atomic) Elements are read from here + size_t localTail; // An uncontended shadow copy of tail, owned by the consumer + + char cachelineFiller0[MOODYCAMEL_CACHE_LINE_SIZE - sizeof(weak_atomic) - sizeof(size_t)]; + weak_atomic tail; // (Atomic) Elements are enqueued here + size_t localFront; + + char cachelineFiller1[MOODYCAMEL_CACHE_LINE_SIZE - sizeof(weak_atomic) - sizeof(size_t)]; // next isn't very contended, but we don't want it on the same cache line as tail (which is) + weak_atomic next; // (Atomic) + + char* data; // Contents (on heap) are aligned to T's alignment + + const size_t sizeMask; + + + // size must be a power of two (and greater than 0) + AE_NO_TSAN Block(size_t const& _size, char* _rawThis, char* _data) + : front(0UL), localTail(0), tail(0UL), localFront(0), next(nullptr), data(_data), sizeMask(_size - 1), rawThis(_rawThis) + { + } + + private: + // C4512 - Assignment operator could not be generated + Block& operator=(Block const&); + + public: + char* rawThis; + }; + + + static Block* make_block(size_t capacity) AE_NO_TSAN + { + // Allocate enough memory for the block itself, as well as all the elements it will contain + auto size = sizeof(Block) + std::alignment_of::value - 1; + size += sizeof(T) * capacity + std::alignment_of::value - 1; + auto newBlockRaw = static_cast(std::malloc(size)); + if (newBlockRaw == nullptr) { + return nullptr; + } + + auto newBlockAligned = align_for(newBlockRaw); + auto newBlockData = align_for(newBlockAligned + sizeof(Block)); + return new (newBlockAligned) Block(capacity, newBlockRaw, newBlockData); + } + +private: + weak_atomic frontBlock; // (Atomic) Elements are dequeued from this block + + char cachelineFiller[MOODYCAMEL_CACHE_LINE_SIZE - sizeof(weak_atomic)]; + weak_atomic tailBlock; // (Atomic) Elements are enqueued to this block + + size_t largestBlockSize; + +#ifndef NDEBUG + weak_atomic enqueuing; + mutable weak_atomic dequeuing; +#endif +}; + +// Like ReaderWriterQueue, but also providees blocking operations +template +class BlockingReaderWriterQueue +{ +private: + typedef ::moodycamel::ReaderWriterQueue ReaderWriterQueue; + +public: + explicit BlockingReaderWriterQueue(size_t size = 15) AE_NO_TSAN + : inner(size), sema(new spsc_sema::LightweightSemaphore()) + { } + + BlockingReaderWriterQueue(BlockingReaderWriterQueue&& other) AE_NO_TSAN + : inner(std::move(other.inner)), sema(std::move(other.sema)) + { } + + BlockingReaderWriterQueue& operator=(BlockingReaderWriterQueue&& other) AE_NO_TSAN + { + std::swap(sema, other.sema); + std::swap(inner, other.inner); + return *this; + } + + + // Enqueues a copy of element if there is room in the queue. + // Returns true if the element was enqueued, false otherwise. + // Does not allocate memory. + AE_FORCEINLINE bool try_enqueue(T const& element) AE_NO_TSAN + { + if (inner.try_enqueue(element)) { + sema->signal(); + return true; + } + return false; + } + + // Enqueues a moved copy of element if there is room in the queue. + // Returns true if the element was enqueued, false otherwise. + // Does not allocate memory. + AE_FORCEINLINE bool try_enqueue(T&& element) AE_NO_TSAN + { + if (inner.try_enqueue(std::forward(element))) { + sema->signal(); + return true; + } + return false; + } + +#if MOODYCAMEL_HAS_EMPLACE + // Like try_enqueue() but with emplace semantics (i.e. construct-in-place). + template + AE_FORCEINLINE bool try_emplace(Args&&... args) AE_NO_TSAN + { + if (inner.try_emplace(std::forward(args)...)) { + sema->signal(); + return true; + } + return false; + } +#endif + + + // Enqueues a copy of element on the queue. + // Allocates an additional block of memory if needed. + // Only fails (returns false) if memory allocation fails. + AE_FORCEINLINE bool enqueue(T const& element) AE_NO_TSAN + { + if (inner.enqueue(element)) { + sema->signal(); + return true; + } + return false; + } + + // Enqueues a moved copy of element on the queue. + // Allocates an additional block of memory if needed. + // Only fails (returns false) if memory allocation fails. + AE_FORCEINLINE bool enqueue(T&& element) AE_NO_TSAN + { + if (inner.enqueue(std::forward(element))) { + sema->signal(); + return true; + } + return false; + } + +#if MOODYCAMEL_HAS_EMPLACE + // Like enqueue() but with emplace semantics (i.e. construct-in-place). + template + AE_FORCEINLINE bool emplace(Args&&... args) AE_NO_TSAN + { + if (inner.emplace(std::forward(args)...)) { + sema->signal(); + return true; + } + return false; + } +#endif + + + // Attempts to dequeue an element; if the queue is empty, + // returns false instead. If the queue has at least one element, + // moves front to result using operator=, then returns true. + template + bool try_dequeue(U& result) AE_NO_TSAN + { + if (sema->tryWait()) { + bool success = inner.try_dequeue(result); + assert(success); + AE_UNUSED(success); + return true; + } + return false; + } + + + // Attempts to dequeue an element; if the queue is empty, + // waits until an element is available, then dequeues it. + template + void wait_dequeue(U& result) AE_NO_TSAN + { + while (!sema->wait()); + bool success = inner.try_dequeue(result); + AE_UNUSED(result); + assert(success); + AE_UNUSED(success); + } + + + // Attempts to dequeue an element; if the queue is empty, + // waits until an element is available up to the specified timeout, + // then dequeues it and returns true, or returns false if the timeout + // expires before an element can be dequeued. + // Using a negative timeout indicates an indefinite timeout, + // and is thus functionally equivalent to calling wait_dequeue. + template + bool wait_dequeue_timed(U& result, std::int64_t timeout_usecs) AE_NO_TSAN + { + if (!sema->wait(timeout_usecs)) { + return false; + } + bool success = inner.try_dequeue(result); + AE_UNUSED(result); + assert(success); + AE_UNUSED(success); + return true; + } + + +#if __cplusplus > 199711L || _MSC_VER >= 1700 + // Attempts to dequeue an element; if the queue is empty, + // waits until an element is available up to the specified timeout, + // then dequeues it and returns true, or returns false if the timeout + // expires before an element can be dequeued. + // Using a negative timeout indicates an indefinite timeout, + // and is thus functionally equivalent to calling wait_dequeue. + template + inline bool wait_dequeue_timed(U& result, std::chrono::duration const& timeout) AE_NO_TSAN + { + return wait_dequeue_timed(result, std::chrono::duration_cast(timeout).count()); + } +#endif + + + // Returns a pointer to the front element in the queue (the one that + // would be removed next by a call to `try_dequeue` or `pop`). If the + // queue appears empty at the time the method is called, nullptr is + // returned instead. + // Must be called only from the consumer thread. + AE_FORCEINLINE T* peek() const AE_NO_TSAN + { + return inner.peek(); + } + + // Removes the front element from the queue, if any, without returning it. + // Returns true on success, or false if the queue appeared empty at the time + // `pop` was called. + AE_FORCEINLINE bool pop() AE_NO_TSAN + { + if (sema->tryWait()) { + bool result = inner.pop(); + assert(result); + AE_UNUSED(result); + return true; + } + return false; + } + + // Returns the approximate number of items currently in the queue. + // Safe to call from both the producer and consumer threads. + AE_FORCEINLINE size_t size_approx() const AE_NO_TSAN + { + return sema->availableApprox(); + } + + // Returns the total number of items that could be enqueued without incurring + // an allocation when this queue is empty. + // Safe to call from both the producer and consumer threads. + // + // NOTE: The actual capacity during usage may be different depending on the consumer. + // If the consumer is removing elements concurrently, the producer cannot add to + // the block the consumer is removing from until it's completely empty, except in + // the case where the producer was writing to the same block the consumer was + // reading from the whole time. + AE_FORCEINLINE size_t max_capacity() const { + return inner.max_capacity(); + } + +private: + // Disable copying & assignment + BlockingReaderWriterQueue(BlockingReaderWriterQueue const&) { } + BlockingReaderWriterQueue& operator=(BlockingReaderWriterQueue const&) { } + +private: + ReaderWriterQueue inner; + std::unique_ptr sema; +}; + +} // end namespace moodycamel + +#ifdef AE_VCPP +#pragma warning(pop) +#endif diff --git a/moodycamel/samples.md b/moodycamel/samples.md old mode 100644 new mode 100755 index 00905ab01..cd73c3b02 --- a/moodycamel/samples.md +++ b/moodycamel/samples.md @@ -7,114 +7,114 @@ extra speed. ## Hello queue +```C++ +ConcurrentQueue q; - ConcurrentQueue q; - - for (int i = 0; i != 123; ++i) - q.enqueue(i); - - int item; - for (int i = 0; i != 123; ++i) { - q.try_dequeue(item); - assert(item == i); - } +for (int i = 0; i != 123; ++i) + q.enqueue(i); +int item; +for (int i = 0; i != 123; ++i) { + q.try_dequeue(item); + assert(item == i); +} +``` ## Hello concurrency Basic example of how to use the queue from multiple threads, with no particular goal (i.e. it does nothing, but in an instructive way). - - ConcurrentQueue q; - int dequeued[100] = { 0 }; - std::thread threads[20]; - - // Producers - for (int i = 0; i != 10; ++i) { - threads[i] = std::thread([&](int i) { - for (int j = 0; j != 10; ++j) { - q.enqueue(i * 10 + j); - } - }, i); - } - - // Consumers - for (int i = 10; i != 20; ++i) { - threads[i] = std::thread([&]() { - int item; - for (int j = 0; j != 20; ++j) { - if (q.try_dequeue(item)) { - ++dequeued[item]; - } +```C++ +ConcurrentQueue q; +int dequeued[100] = { 0 }; +std::thread threads[20]; + +// Producers +for (int i = 0; i != 10; ++i) { + threads[i] = std::thread([&](int i) { + for (int j = 0; j != 10; ++j) { + q.enqueue(i * 10 + j); + } + }, i); +} + +// Consumers +for (int i = 10; i != 20; ++i) { + threads[i] = std::thread([&]() { + int item; + for (int j = 0; j != 20; ++j) { + if (q.try_dequeue(item)) { + ++dequeued[item]; } - }); - } - - // Wait for all threads - for (int i = 0; i != 20; ++i) { - threads[i].join(); - } - - // Collect any leftovers (could be some if e.g. consumers finish before producers) - int item; - while (q.try_dequeue(item)) { - ++dequeued[item]; - } - - // Make sure everything went in and came back out! - for (int i = 0; i != 100; ++i) { - assert(dequeued[i] == 1); - } - + } + }); +} + +// Wait for all threads +for (int i = 0; i != 20; ++i) { + threads[i].join(); +} + +// Collect any leftovers (could be some if e.g. consumers finish before producers) +int item; +while (q.try_dequeue(item)) { + ++dequeued[item]; +} + +// Make sure everything went in and came back out! +for (int i = 0; i != 100; ++i) { + assert(dequeued[i] == 1); +} +``` ## Bulk up Same as previous example, but runs faster. - - ConcurrentQueue q; - int dequeued[100] = { 0 }; - std::thread threads[20]; - - // Producers - for (int i = 0; i != 10; ++i) { - threads[i] = std::thread([&](int i) { - int items[10]; - for (int j = 0; j != 10; ++j) { - items[j] = i * 10 + j; - } - q.enqueue_bulk(items, 10); - }, i); - } - - // Consumers - for (int i = 10; i != 20; ++i) { - threads[i] = std::thread([&]() { - int items[20]; - for (std::size_t count = q.try_dequeue_bulk(items, 20); count != 0; --count) { - ++dequeued[items[count - 1]]; - } - }); - } - - // Wait for all threads - for (int i = 0; i != 20; ++i) { - threads[i].join(); - } - - // Collect any leftovers (could be some if e.g. consumers finish before producers) - int items[10]; - std::size_t count; - while ((count = q.try_dequeue_bulk(items, 10)) != 0) { - for (std::size_t i = 0; i != count; ++i) { - ++dequeued[items[i]]; +```C++ +ConcurrentQueue q; +int dequeued[100] = { 0 }; +std::thread threads[20]; + +// Producers +for (int i = 0; i != 10; ++i) { + threads[i] = std::thread([&](int i) { + int items[10]; + for (int j = 0; j != 10; ++j) { + items[j] = i * 10 + j; } + q.enqueue_bulk(items, 10); + }, i); +} + +// Consumers +for (int i = 10; i != 20; ++i) { + threads[i] = std::thread([&]() { + int items[20]; + for (std::size_t count = q.try_dequeue_bulk(items, 20); count != 0; --count) { + ++dequeued[items[count - 1]]; + } + }); +} + +// Wait for all threads +for (int i = 0; i != 20; ++i) { + threads[i].join(); +} + +// Collect any leftovers (could be some if e.g. consumers finish before producers) +int items[10]; +std::size_t count; +while ((count = q.try_dequeue_bulk(items, 10)) != 0) { + for (std::size_t i = 0; i != count; ++i) { + ++dequeued[items[i]]; } - - // Make sure everything went in and came back out! - for (int i = 0; i != 100; ++i) { - assert(dequeued[i] == 1); - } +} +// Make sure everything went in and came back out! +for (int i = 0; i != 100; ++i) { + assert(dequeued[i] == 1); +} +``` ## Producer/consumer model (simultaneous) @@ -122,46 +122,46 @@ In this model, one set of threads is producing items, and the other is consuming them concurrently until all of them have been consumed. The counters are required to ensure that all items eventually get consumed. - - ConcurrentQueue q; - const int ProducerCount = 8; - const int ConsumerCount = 8; - std::thread producers[ProducerCount]; - std::thread consumers[ConsumerCount]; - std::atomic doneProducers(0); - std::atomic doneConsumers(0); - for (int i = 0; i != ProducerCount; ++i) { - producers[i] = std::thread([&]() { - while (produce) { - q.enqueue(produceItem()); +```C++ +ConcurrentQueue q; +const int ProducerCount = 8; +const int ConsumerCount = 8; +std::thread producers[ProducerCount]; +std::thread consumers[ConsumerCount]; +std::atomic doneProducers(0); +std::atomic doneConsumers(0); +for (int i = 0; i != ProducerCount; ++i) { + producers[i] = std::thread([&]() { + while (produce) { + q.enqueue(produceItem()); + } + doneProducers.fetch_add(1, std::memory_order_release); + }); +} +for (int i = 0; i != ConsumerCount; ++i) { + consumers[i] = std::thread([&]() { + Item item; + bool itemsLeft; + do { + // It's important to fence (if the producers have finished) *before* dequeueing + itemsLeft = doneProducers.load(std::memory_order_acquire) != ProducerCount; + while (q.try_dequeue(item)) { + itemsLeft = true; + consumeItem(item); } - doneProducers.fetch_add(1, std::memory_order_release); - }); - } - for (int i = 0; i != ConsumerCount; ++i) { - consumers[i] = std::thread([&]() { - Item item; - bool itemsLeft; - do { - // It's important to fence (if the producers have finished) *before* dequeueing - itemsLeft = doneProducers.load(std::memory_order_acquire) != ProducerCount; - while (q.try_dequeue(item)) { - itemsLeft = true; - consumeItem(item); - } - } while (itemsLeft || doneConsumers.fetch_add(1, std::memory_order_acq_rel) + 1 == ConsumerCount); - // The condition above is a bit tricky, but it's necessary to ensure that the - // last consumer sees the memory effects of all the other consumers before it - // calls try_dequeue for the last time - }); - } - for (int i = 0; i != ProducerCount; ++i) { - producers[i].join(); - } - for (int i = 0; i != ConsumerCount; ++i) { - consumers[i].join(); - } - + } while (itemsLeft || doneConsumers.fetch_add(1, std::memory_order_acq_rel) + 1 == ConsumerCount); + // The condition above is a bit tricky, but it's necessary to ensure that the + // last consumer sees the memory effects of all the other consumers before it + // calls try_dequeue for the last time + }); +} +for (int i = 0; i != ProducerCount; ++i) { + producers[i].join(); +} +for (int i = 0; i != ConsumerCount; ++i) { + consumers[i].join(); +} +``` ## Producer/consumer model (simultaneous, blocking) The blocking version is different, since either the number of elements being produced needs @@ -169,72 +169,72 @@ to be known ahead of time, or some other coordination is required to tell the co to stop calling wait_dequeue (not shown here). This is necessary because otherwise a consumer could end up blocking forever -- and destroying a queue while a consumer is blocking on it leads to undefined behaviour. - - BlockingConcurrentQueue q; - const int ProducerCount = 8; - const int ConsumerCount = 8; - std::thread producers[ProducerCount]; - std::thread consumers[ConsumerCount]; - std::atomic promisedElementsRemaining(ProducerCount * 1000); - for (int i = 0; i != ProducerCount; ++i) { - producers[i] = std::thread([&]() { - for (int j = 0; j != 1000; ++j) { - q.enqueue(produceItem()); - } - }); - } - for (int i = 0; i != ConsumerCount; ++i) { - consumers[i] = std::thread([&]() { - Item item; - while (promisedElementsRemaining.fetch_sub(1, std::memory_order_relaxed)) { - q.wait_dequeue(item); - consumeItem(item); - } - }); - } - for (int i = 0; i != ProducerCount; ++i) { - producers[i].join(); - } - for (int i = 0; i != ConsumerCount; ++i) { - consumers[i].join(); - } - +```C++ +BlockingConcurrentQueue q; +const int ProducerCount = 8; +const int ConsumerCount = 8; +std::thread producers[ProducerCount]; +std::thread consumers[ConsumerCount]; +std::atomic promisedElementsRemaining(ProducerCount * 1000); +for (int i = 0; i != ProducerCount; ++i) { + producers[i] = std::thread([&]() { + for (int j = 0; j != 1000; ++j) { + q.enqueue(produceItem()); + } + }); +} +for (int i = 0; i != ConsumerCount; ++i) { + consumers[i] = std::thread([&]() { + Item item; + while (promisedElementsRemaining.fetch_sub(1, std::memory_order_relaxed)) { + q.wait_dequeue(item); + consumeItem(item); + } + }); +} +for (int i = 0; i != ProducerCount; ++i) { + producers[i].join(); +} +for (int i = 0; i != ConsumerCount; ++i) { + consumers[i].join(); +} +``` ## Producer/consumer model (separate stages) - - ConcurrentQueue q; - - // Production stage - std::thread threads[8]; - for (int i = 0; i != 8; ++i) { - threads[i] = std::thread([&]() { - while (produce) { - q.enqueue(produceItem()); +```C++ +ConcurrentQueue q; + +// Production stage +std::thread threads[8]; +for (int i = 0; i != 8; ++i) { + threads[i] = std::thread([&]() { + while (produce) { + q.enqueue(produceItem()); + } + }); +} +for (int i = 0; i != 8; ++i) { + threads[i].join(); +} + +// Consumption stage +std::atomic doneConsumers(0); +for (int i = 0; i != 8; ++i) { + threads[i] = std::thread([&]() { + Item item; + do { + while (q.try_dequeue(item)) { + consumeItem(item); } - }); - } - for (int i = 0; i != 8; ++i) { - threads[i].join(); - } - - // Consumption stage - std::atomic doneConsumers(0); - for (int i = 0; i != 8; ++i) { - threads[i] = std::thread([&]() { - Item item; - do { - while (q.try_dequeue(item)) { - consumeItem(item); - } - // Loop again one last time if we're the last producer (with the acquired - // memory effects of the other producers): - } while (doneConsumers.fetch_add(1, std::memory_order_acq_rel) + 1 == 8); - }); - } - for (int i = 0; i != 8; ++i) { - threads[i].join(); - } - + // Loop again one last time if we're the last producer (with the acquired + // memory effects of the other producers): + } while (doneConsumers.fetch_add(1, std::memory_order_acq_rel) + 1 == 8); + }); +} +for (int i = 0; i != 8; ++i) { + threads[i].join(); +} +``` Note that there's no point trying to use the blocking queue with this model, since there's no need to use the `wait` methods (all the elements are produced before any are consumed), and hence the complexity would be the same but with additional overhead. @@ -245,80 +245,80 @@ are consumed), and hence the complexity would be the same but with additional ov If you don't know what threads will be using the queue in advance, you can't really declare any long-term tokens. The obvious solution is to use the implicit methods (that don't take any tokens): - - // A pool of 'Something' objects that can be safely accessed - // from any thread - class SomethingPool +```C++ +// A pool of 'Something' objects that can be safely accessed +// from any thread +class SomethingPool +{ +public: + Something getSomething() { - public: - Something getSomething() - { - Something obj; - queue.try_dequeue(obj); - - // If the dequeue succeeded, obj will be an object from the - // thread pool, otherwise it will be the default-constructed - // object as declared above - return obj; - } - - void recycleSomething(Something&& obj) - { - queue.enqueue(std::move(obj)); - } - }; + Something obj; + queue.try_dequeue(obj); + // If the dequeue succeeded, obj will be an object from the + // thread pool, otherwise it will be the default-constructed + // object as declared above + return obj; + } + + void recycleSomething(Something&& obj) + { + queue.enqueue(std::move(obj)); + } +}; +``` ## Threadpool task queue +```C++ +BlockingConcurrentQueue q; - BlockingConcurrentQueue q; - - // To create a task from any thread: - q.enqueue(...); - - // On threadpool threads: - Task task; - while (true) { - q.wait_dequeue(task); - - // Process task... - } +// To create a task from any thread: +q.enqueue(...); + +// On threadpool threads: +Task task; +while (true) { + q.wait_dequeue(task); + // Process task... +} +``` ## Multithreaded game loop +```C++ +BlockingConcurrentQueue q; +std::atomic pendingTasks(0); - BlockingConcurrentQueue q; - std::atomic pendingTasks(0); - - // On threadpool threads: - Task task; - while (true) { - q.wait_dequeue(task); - - // Process task... - - pendingTasks.fetch_add(-1, std::memory_order_release); - } - - // Whenever a new task needs to be processed for the frame: - pendingTasks.fetch_add(1, std::memory_order_release); - q.enqueue(...); - - // To wait for all the frame's tasks to complete before rendering: - while (pendingTasks.load(std::memory_order_acquire) != 0) +// On threadpool threads: +Task task; +while (true) { + q.wait_dequeue(task); + + // Process task... + + pendingTasks.fetch_add(-1, std::memory_order_release); +} + +// Whenever a new task needs to be processed for the frame: +pendingTasks.fetch_add(1, std::memory_order_release); +q.enqueue(...); + +// To wait for all the frame's tasks to complete before rendering: +while (pendingTasks.load(std::memory_order_acquire) != 0) + continue; + +// Alternatively you could help out the thread pool while waiting: +while (pendingTasks.load(std::memory_order_acquire) != 0) { + if (!q.try_dequeue(task)) { continue; - - // Alternatively you could help out the thread pool while waiting: - while (pendingTasks.load(std::memory_order_acquire) != 0) { - if (!q.try_dequeue(task)) { - continue; - } - - // Process task... - - pendingTasks.fetch_add(-1, std::memory_order_release); } - + + // Process task... + + pendingTasks.fetch_add(-1, std::memory_order_release); +} +``` ## Pump until empty @@ -328,42 +328,42 @@ to ensure that the memory effects of any enqueue operations you wish to see on the dequeue thread are visible (i.e. if you're waiting for a certain set of elements, you need to use memory fences to ensure that those elements are visible to the dequeue thread after they've been enqueued). - - ConcurrentQueue q; - - // Single-threaded pumping: - Item item; - while (q.try_dequeue(item)) { - // Process item... - } - // q is guaranteed to be empty here, unless there is another thread enqueueing still or - // there was another thread dequeueing at one point and its memory effects have not - // yet been propagated to this thread. - - // Multi-threaded pumping: - std::thread threads[8]; - std::atomic doneConsumers(0); - for (int i = 0; i != 8; ++i) { - threads[i] = std::thread([&]() { - Item item; - do { - while (q.try_dequeue(item)) { - // Process item... - } - } while (doneConsumers.fetch_add(1, std::memory_order_acq_rel) + 1 == 8); - // If there are still enqueue operations happening on other threads, - // then the queue may not be empty at this point. However, if all enqueue - // operations completed before we finished pumping (and the propagation of - // their memory effects too), and all dequeue operations apart from those - // our threads did above completed before we finished pumping (and the - // propagation of their memory effects too), then the queue is guaranteed - // to be empty at this point. - }); - } - for (int i = 0; i != 8; ++i) { - threads[i].join(); - } - +```C++ +ConcurrentQueue q; + +// Single-threaded pumping: +Item item; +while (q.try_dequeue(item)) { + // Process item... +} +// q is guaranteed to be empty here, unless there is another thread enqueueing still or +// there was another thread dequeueing at one point and its memory effects have not +// yet been propagated to this thread. + +// Multi-threaded pumping: +std::thread threads[8]; +std::atomic doneConsumers(0); +for (int i = 0; i != 8; ++i) { + threads[i] = std::thread([&]() { + Item item; + do { + while (q.try_dequeue(item)) { + // Process item... + } + } while (doneConsumers.fetch_add(1, std::memory_order_acq_rel) + 1 == 8); + // If there are still enqueue operations happening on other threads, + // then the queue may not be empty at this point. However, if all enqueue + // operations completed before we finished pumping (and the propagation of + // their memory effects too), and all dequeue operations apart from those + // our threads did above completed before we finished pumping (and the + // propagation of their memory effects too), then the queue is guaranteed + // to be empty at this point. + }); +} +for (int i = 0; i != 8; ++i) { + threads[i].join(); +} +``` ## Wait for a queue to become empty (without dequeueing)