Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update our encryption to match changes in Facebook sources #18

Open
wants to merge 59 commits into
base: stardog/develop
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
8522d9c
Prepare FragmentedRangeTombstoneIterator for use in compaction (#4740)
abhimadan Dec 11, 2018
96de211
Add compaction logic to RangeDelAggregatorV2 (#4758)
abhimadan Dec 17, 2018
33564d2
Remove v1 RangeDelAggregator (#4778)
abhimadan Dec 18, 2018
de0891e
Fix unused member compile error
abhimadan Dec 18, 2018
8a643b7
Detect if Jemalloc is linked with the binary (#4844)
Jan 4, 2019
ec43385
Enable checkpoint of read-only db (#4681)
riversand963 Dec 8, 2018
663d24f
Improve flushing multiple column families (#4708)
riversand963 Dec 13, 2018
e265e08
Avoid switching empty memtable in certain cases (#4792)
riversand963 Dec 19, 2018
35c950a
Refactor atomic flush result installation to MANIFEST (#4791)
riversand963 Jan 4, 2019
97773d0
Update HISTORY.md
riversand963 Jan 7, 2019
e78f5cf
Fix point lookup on range tombstone sentinel endpoint (#4829)
ajkr Jan 4, 2019
3bcc312
Initialize two members in PerfContext (#4859)
riversand963 Jan 9, 2019
4eeb1bf
Bump version to 5.18.1
riversand963 Jan 10, 2019
9ae0528
Use chrono::time_point instead of time_t (#4868)
riversand963 Jan 16, 2019
35c05bc
Deleting Blob files also goes through SstFileManager (#4904)
siying Jan 23, 2019
53f760b
Always delete Blob DB files in the background (#4928)
Jan 29, 2019
acba14b
Make a copy of MutableCFOptions to avoid race condition (#4876)
riversand963 Jan 12, 2019
65b2298
Use correct FileMeta for atomic flush result install (#4932)
riversand963 Jan 31, 2019
a1774dd
Bump version to 5.18.2
sagar0 Jan 31, 2019
b7434c2
Properly set upper bound of subcompaction output (#4879) (#4898)
Feb 5, 2019
641fae6
update history and bump version
ajkr Feb 11, 2019
5e2f968
Revert "Core-local statistics"
leventov Jun 4, 2018
4e6b92c
Update cmake to not build tests when not in Debug
toktarev Jun 7, 2018
05b54b1
Column family destructor should print column family name on assertion…
toktarev Jul 4, 2018
b1cc1a8
Use the C++ steady_clock on Mac OSX, rather than using the Mach kerne…
Aug 1, 2018
33659a3
Post cherry-pick fixes
toktarev Mar 13, 2019
fdb84c8
Upgrade to 5.18.3
toktarev Mar 19, 2019
3182761
Working Bazel build
Sep 29, 2018
2520fc9
Update OSX defines
Oct 11, 2018
4704cbe
Build with toolchain
Oct 15, 2018
e9a1ddf
Fixed link issue
Oct 15, 2018
30b49bc
Windows is not POSIX
Oct 25, 2018
34703d9
Exclude benchmarking executable range_del_aggregator_bench.cc from db…
Dec 6, 2018
f4644de
Remove dependency on gtest_main as that seems to cause some bad depen…
Oct 28, 2018
0b86b91
Include jemalloc_helper.h only if ROCKSDB_JEMALLOC is enabled
toktarev Mar 19, 2019
98bf022
Added jemalloc_helper.h as a dependency
toktarev Mar 21, 2019
feb5490
Make content_flags_ to be available for public API
toktarev Apr 16, 2019
3e7faa1
Merge branch 'v5.18.3' of github.com:stardog-union/rocksdb into v5.18.3
Apr 30, 2019
41129d5
Turned off performance measurement by default
Sep 26, 2019
5636fba
clean include prefix from rocksdb include paths to fix pragma once pr…
Jan 21, 2020
75c14f9
force win_jemalloc out of the bazel build
Jan 22, 2020
8847d69
wow, we do build these utility files that have the same stupid includ…
Jan 22, 2020
c164550
additional deps to help Windows build
Jan 27, 2020
4556837
mac os cross build suddenly needs std::move definition? why?
Feb 11, 2020
6def808
Adding "_WIN32_WINNT=_WIN32_WINNT_VISTA" to compile defines
Mar 19, 2020
c6063a4
Feature: AES CTR encryption (#6)
matthewvon May 28, 2020
a7d2e8d
fill in missing CompactionJobInfo member. (#8)
matthewvon Jun 5, 2020
5cb1595
default skiplist height same as original leveldb which had 4M write b…
matthewvon Jun 12, 2020
844661e
Porting Facebook required changes back to Stardog (#10)
matthewvon Jun 23, 2020
2c35685
previous PR missed ifdefs within library_loader stuff and switch to u…
matthewvon Jun 30, 2020
4569cef
Backport changes to encryption that Facebook required. (#12)
matthewvon Jul 9, 2020
6acb712
Feature/mv encrypt 2 (#15)
matthewvon Aug 21, 2020
c46e40d
make SetKeys public since now rwlock protected. (#16)
matthewvon Aug 28, 2020
529487e
make mutex mutable for const function
matthewvon Sep 1, 2020
174ccbd
Merge pull request #17 from stardog-union/mv-channel-mutable-lock
jbalint Sep 1, 2020
7ecb246
backport Facebook changes to env_encryption ... then update Stardog o…
matthewvon Sep 6, 2020
ff46be7
update unit test (and fix what it found broken)
matthewvon Sep 7, 2020
743b314
add a paranoid test for all potential partial block starts of Encrypt
matthewvon Sep 8, 2020
a894db0
have paranoid test execute across two offset (32 bytes) instead of ju…
matthewvon Sep 8, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Detect if Jemalloc is linked with the binary (#4844)
Summary:
Declare Jemalloc non-standard APIs as weak symbols, so that if Jemalloc is linked with the binary, these symbols will be replaced by Jemalloc's, otherwise they will be nullptr. This is similar to how folly detect jemalloc, but we assume the main program use jemalloc as long as jemalloc is linked: https://github.com/facebook/folly/blob/master/folly/memory/Malloc.h#L147
Pull Request resolved: facebook/rocksdb#4844

Differential Revision: D13574934

Pulled By: yiwu-arbug

fbshipit-source-id: 7ea871beb1be7d5a1259cc38f9b78078793db2db
Yi Wu committed Jan 4, 2019

Verified

This commit was signed with the committer’s verified signature.
commit 8a643b70fd5511cc16f8398148518ba7444b14ae
8 changes: 3 additions & 5 deletions TARGETS
Original file line number Diff line number Diff line change
@@ -67,13 +67,11 @@ is_opt_mode = build_mode.startswith("opt")
if is_opt_mode:
rocksdb_compiler_flags.append("-DNDEBUG")

default_allocator = read_config("fbcode", "default_allocator")

sanitizer = read_config("fbcode", "sanitizer")

# Let RocksDB aware of jemalloc existence.
# Do not enable it if sanitizer presents.
if is_opt_mode and default_allocator.startswith("jemalloc") and sanitizer == "":
# Do not enable jemalloc if sanitizer presents. RocksDB will further detect
# whether the binary is linked with jemalloc at runtime.
if sanitizer == "":
rocksdb_compiler_flags.append("-DROCKSDB_JEMALLOC")
rocksdb_external_deps.append(("jemalloc", None, "headers"))

8 changes: 3 additions & 5 deletions buckifier/targets_cfg.py
Original file line number Diff line number Diff line change
@@ -71,13 +71,11 @@
if is_opt_mode:
rocksdb_compiler_flags.append("-DNDEBUG")
default_allocator = read_config("fbcode", "default_allocator")
sanitizer = read_config("fbcode", "sanitizer")
# Let RocksDB aware of jemalloc existence.
# Do not enable it if sanitizer presents.
if is_opt_mode and default_allocator.startswith("jemalloc") and sanitizer == "":
# Do not enable jemalloc if sanitizer presents. RocksDB will further detect
# whether the binary is linked with jemalloc at runtime.
if sanitizer == "":
rocksdb_compiler_flags.append("-DROCKSDB_JEMALLOC")
rocksdb_external_deps.append(("jemalloc", None, "headers"))
"""
17 changes: 8 additions & 9 deletions db/malloc_stats.cc
Original file line number Diff line number Diff line change
@@ -13,17 +13,16 @@
#include <memory>
#include <string.h>

#include "port/jemalloc_helper.h"


namespace rocksdb {

#ifdef ROCKSDB_JEMALLOC
#ifdef __FreeBSD__
#include <malloc_np.h>
#else
#include "jemalloc/jemalloc.h"

#ifdef JEMALLOC_NO_RENAME
#define malloc_stats_print je_malloc_stats_print
#endif
#endif

typedef struct {
char* cur;
@@ -41,10 +40,10 @@ static void GetJemallocStatus(void* mstat_arg, const char* status) {
snprintf(mstat->cur, buf_size, "%s", status);
mstat->cur += status_len;
}
#endif // ROCKSDB_JEMALLOC

#ifdef ROCKSDB_JEMALLOC
void DumpMallocStats(std::string* stats) {
if (!HasJemalloc()) {
return;
}
MallocStatus mstat;
const unsigned int kMallocStatusLen = 1000000;
std::unique_ptr<char[]> buf{new char[kMallocStatusLen + 1]};
@@ -56,5 +55,5 @@ void DumpMallocStats(std::string* stats) {
#else
void DumpMallocStats(std::string*) {}
#endif // ROCKSDB_JEMALLOC
}
} // namespace rocksdb
#endif // !ROCKSDB_LITE
49 changes: 49 additions & 0 deletions port/jemalloc_helper.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).

#pragma once

#ifdef ROCKSDB_JEMALLOC
#ifdef __FreeBSD__
#include <malloc_np.h>
#else
#include <jemalloc/jemalloc.h>
#endif

// Declare non-standard jemalloc APIs as weak symbols. We can null-check these
// symbols to detect whether jemalloc is linked with the binary.
extern "C" void* mallocx(size_t, int) __attribute__((__weak__));
extern "C" void* rallocx(void*, size_t, int) __attribute__((__weak__));
extern "C" size_t xallocx(void*, size_t, size_t, int) __attribute__((__weak__));
extern "C" size_t sallocx(const void*, int) __attribute__((__weak__));
extern "C" void dallocx(void*, int) __attribute__((__weak__));
extern "C" void sdallocx(void*, size_t, int) __attribute__((__weak__));
extern "C" size_t nallocx(size_t, int) __attribute__((__weak__));
extern "C" int mallctl(const char*, void*, size_t*, void*, size_t)
__attribute__((__weak__));
extern "C" int mallctlnametomib(const char*, size_t*, size_t*)
__attribute__((__weak__));
extern "C" int mallctlbymib(const size_t*, size_t, void*, size_t*, void*,
size_t) __attribute__((__weak__));
extern "C" void malloc_stats_print(void (*)(void*, const char*), void*,
const char*) __attribute__((__weak__));
extern "C" size_t malloc_usable_size(JEMALLOC_USABLE_SIZE_CONST void*)
JEMALLOC_CXX_THROW __attribute__((__weak__));

// Check if Jemalloc is linked with the binary. Note the main program might be
// using a different memory allocator even this method return true.
// It is loosely based on folly::usingJEMalloc(), minus the check that actually
// allocate memory and see if it is through jemalloc, to handle the dlopen()
// case:
// https://github.com/facebook/folly/blob/76cf8b5841fb33137cfbf8b224f0226437c855bc/folly/memory/Malloc.h#L147
static inline bool HasJemalloc() {
return mallocx != nullptr && rallocx != nullptr && xallocx != nullptr &&
sallocx != nullptr && dallocx != nullptr && sdallocx != nullptr &&
nallocx != nullptr && mallctl != nullptr &&
mallctlnametomib != nullptr && mallctlbymib != nullptr &&
malloc_stats_print != nullptr && malloc_usable_size != nullptr;
}

#endif // ROCKSDB_JEMALLOC
10 changes: 7 additions & 3 deletions util/jemalloc_nodump_allocator.cc
Original file line number Diff line number Diff line change
@@ -133,12 +133,16 @@ Status NewJemallocNodumpAllocator(
JemallocAllocatorOptions& options,
std::shared_ptr<MemoryAllocator>* memory_allocator) {
*memory_allocator = nullptr;
#ifndef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
(void) options;
return Status::NotSupported(
Status unsupported = Status::NotSupported(
"JemallocNodumpAllocator only available with jemalloc version >= 5 "
"and MADV_DONTDUMP is available.");
#ifndef ROCKSDB_JEMALLOC_NODUMP_ALLOCATOR
(void)options;
return unsupported;
#else
if (!HasJemalloc()) {
return unsupported;
}
if (memory_allocator == nullptr) {
return Status::InvalidArgument("memory_allocator must be non-null.");
}
2 changes: 1 addition & 1 deletion util/jemalloc_nodump_allocator.h
Original file line number Diff line number Diff line change
@@ -8,14 +8,14 @@
#include <atomic>
#include <vector>

#include "port/jemalloc_helper.h"
#include "port/port.h"
#include "rocksdb/memory_allocator.h"
#include "util/core_local.h"
#include "util/thread_local.h"

#if defined(ROCKSDB_JEMALLOC) && defined(ROCKSDB_PLATFORM_POSIX)

#include <jemalloc/jemalloc.h>
#include <sys/mman.h>

#if (JEMALLOC_VERSION_MAJOR >= 5) && defined(MADV_DONTDUMP)