Skip to content

Commit

Permalink
align document naming (#4031)
Browse files Browse the repository at this point in the history
* rename cache_management.md into cache.md
* rename encrypt.md into encryption.md
  • Loading branch information
timfeirg authored Aug 31, 2023
1 parent 3a1c28c commit 3e02c31
Show file tree
Hide file tree
Showing 31 changed files with 62 additions and 64 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ If you wanna use JuiceFS in Hadoop, check [Hadoop Java SDK](https://juicefs.com/

- [Redis Best Practices](https://juicefs.com/docs/community/redis_best_practices)
- [How to Setup Object Storage](https://juicefs.com/docs/community/how_to_setup_object_storage)
- [Cache Management](https://juicefs.com/docs/community/cache_management)
- [Cache](https://juicefs.com/docs/community/cache)
- [Fault Diagnosis and Analysis](https://juicefs.com/docs/community/fault_diagnosis_and_analysis)
- [FUSE Mount Options](https://juicefs.com/docs/community/fuse_mount_options)
- [Using JuiceFS on Windows](https://juicefs.com/docs/community/installation#windows)
Expand Down
2 changes: 1 addition & 1 deletion README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ JuiceFS 使用 [Hadoop Java SDK](https://juicefs.com/docs/zh/community/hadoop_ja

- [Redis 最佳实践](https://juicefs.com/docs/zh/community/redis_best_practices)
- [如何设置对象存储](https://juicefs.com/docs/zh/community/how_to_setup_object_storage)
- [缓存管理](https://juicefs.com/docs/zh/community/cache_management)
- [缓存](https://juicefs.com/docs/zh/community/cache)
- [故障诊断和分析](https://juicefs.com/docs/zh/community/fault_diagnosis_and_analysis)
- [FUSE 挂载选项](https://juicefs.com/docs/zh/community/fuse_mount_options)
- [在 Windows 中使用 JuiceFS](https://juicefs.com/docs/zh/community/installation#windows-系统)
Expand Down
2 changes: 1 addition & 1 deletion docs/en/administration/fault_diagnosis_and_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ Metrics description:

- `cpu`: CPU usage of the process.
- `mem`: Physical memory used by the process.
- `buf`: Current [buffer size](../guide/cache_management.md#buffer-size), if this value is constantly close to (or even exceeds) the configured [`--buffer-size`](../reference/command_reference.md#mount), you should increase buffer size or decrease application workload.
- `buf`: Current [buffer size](../guide/cache.md#buffer-size), if this value is constantly close to (or even exceeds) the configured [`--buffer-size`](../reference/command_reference.md#mount), you should increase buffer size or decrease application workload.
- `cache`: Internal metric, ignore this.

#### `fuse`
Expand Down
2 changes: 1 addition & 1 deletion docs/en/administration/metadata_dump_load.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ juicefs config --secret-key xxxxx mysql://user:password@(192.168.1.6:3306)/juice

### Encrypted file system {#encrypted-file-system}

For [encrypted file system](../security/encrypt.md), all data is encrypted before uploading to the object storage, including automatic metadata backups. This is different from the `dump` command, which only output metadata in plain text.
For [encrypted file system](../security/encryption.md), all data is encrypted before uploading to the object storage, including automatic metadata backups. This is different from the `dump` command, which only output metadata in plain text.

For an encrypted file system, it is necessary to additionally set the `JFS_RSA_PASSPHRASE` environment variable and specify the RSA private key and encryption algorithm when restoring the automatically backed-up metadata:

Expand Down
6 changes: 3 additions & 3 deletions docs/en/administration/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ The first issue with slow connection is upload / download timeouts (demonstrated
* Reduce buffer size, e.g. [`--buffer-size=64`](../reference/command_reference.md#mount) or even lower. In a large bandwidth condition, increasing buffer size improves parallel performance. But in a low speed environment, this only makes `flush` operations slow and prone to timeouts.
* Default timeout for GET / PUT requests are 60 seconds, increasing `--get-timeout` and `--put-timeout` may help with read / write timeouts.

In addition, the ["Client Write Cache"](../guide/cache_management.md#writeback) feature needs to be used with caution in low bandwidth environment. Let's briefly go over the JuiceFS Client background job design: every JuiceFS Client runs background jobs by default, one of which is data compaction, and if the client has poor internet speed, it'll drag down performance for the whole system. A worse case is when client write cache is also enabled, compaction results are uploaded too slowly, forcing other clients into a read hang when accessing the affected files:
In addition, the ["Client Write Cache"](../guide/cache.md#writeback) feature needs to be used with caution in low bandwidth environment. Let's briefly go over the JuiceFS Client background job design: every JuiceFS Client runs background jobs by default, one of which is data compaction, and if the client has poor internet speed, it'll drag down performance for the whole system. A worse case is when client write cache is also enabled, compaction results are uploaded too slowly, forcing other clients into a read hang when accessing the affected files:

```text
# While compaction results are slowly being uploaded in low speed clients, read from other clients will hang and eventually fail
Expand All @@ -115,7 +115,7 @@ To avoid this type of issue, we recommend disabling background jobs on low-bandw

In JuiceFS, a typical read amplification manifests as object storage traffic being much larger than JuiceFS Client read speed. For example, JuiceFS Client is reading at 200MiB/s, while S3 traffic grows up to 2GiB/s.

JuiceFS is equipped with the [prefetch mechanism](../guide/cache_management.md#client-read-cache): when reading a block at arbitrary position, the whole block is asynchronously scheduled for download. This is a read optimization enabled by default, but in some cases, this brings read amplification. Once we know this, we can start the diagnose.
JuiceFS is equipped with the [prefetch mechanism](../guide/cache.md#client-read-cache): when reading a block at arbitrary position, the whole block is asynchronously scheduled for download. This is a read optimization enabled by default, but in some cases, this brings read amplification. Once we know this, we can start the diagnose.

We'll collect JuiceFS access log (see [Access log](./fault_diagnosis_and_analysis.md#access-log)) to determine the file system access patterns of our application, and adjust JuiceFS configuration accordingly. Below is a diagnose process in an actual production environment:

Expand Down Expand Up @@ -157,7 +157,7 @@ Studying the access log, it's easy to conclude that our application performs fre

If JuiceFS Client takes up too much memory, you may choose to optimize memory usage using below methods, but note that memory optimization is not free, and each setting adjustment will bring corresponding overhead, please do sufficient testing and verification before adjustment.

* Read/Write buffer size (`--buffer-size`) directly correlate to JuiceFS Client memory usage, using a lower `--buffer-size` will effectively decrease memory usage, but please note that the reduction may also affect the read and write performance. Read more at [Read/Write Buffer](../guide/cache_management.md#buffer-size).
* Read/Write buffer size (`--buffer-size`) directly correlate to JuiceFS Client memory usage, using a lower `--buffer-size` will effectively decrease memory usage, but please note that the reduction may also affect the read and write performance. Read more at [Read/Write Buffer](../guide/cache.md#buffer-size).
* JuiceFS mount client is an Go program, which means you can decrease `GOGC` (default to 100, in percentage) to adopt a more active garbage collection. This inevitably increase CPU usage and may even directly hinder performance. Read more at [Go Runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables).
* If you use self-hosted Ceph RADOS as the data storage of JuiceFS, consider replacing glibc with [TCMalloc](https://google.github.io/tcmalloc), the latter comes with more efficient memory management and may decrease off-heap memory footprint in this scenario.

Expand Down
2 changes: 1 addition & 1 deletion docs/en/development/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -907,7 +907,7 @@ You can configure the compression algorithm (supporting `lz4` and `zstd`) with t

#### Data encryption

The RSA private key can be configured to enable [static data encryption](../security/encrypt.md) when formatting a file system with the `--encrypt-rsa-key <value>` parameter, which allows all data blocks of this file system to be encrypted before uploading to the object storage. The object name is still the same as default, while its content becomes a header plus the result of the data encryption algorithm. The header contains a random seed and the symmetric key used for decryption, and the symmetric key itself is encrypted with the RSA private key. Therefore, it is not allowed to modify the RSA private key in the [file system formatting Information](#setting), otherwise reading existing data will fail.
The RSA private key can be configured to enable [static data encryption](../security/encryption.md) when formatting a file system with the `--encrypt-rsa-key <value>` parameter, which allows all data blocks of this file system to be encrypted before uploading to the object storage. The object name is still the same as default, while its content becomes a header plus the result of the data encryption algorithm. The header contains a random seed and the symmetric key used for decryption, and the symmetric key itself is encrypted with the RSA private key. Therefore, it is not allowed to modify the RSA private key in the [file system formatting Information](#setting), otherwise reading existing data will fail.

:::note
If both compression and encryption are enabled, the original data will be compressed and then encrypted before uploading to the object storage.
Expand Down
4 changes: 2 additions & 2 deletions docs/en/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Read [JuiceFS Internals](development/internals.md) and [Data Processing Flow](in

You could mount JuiceFS with [`--writeback` option](reference/command_reference.md#mount), which will write the small files into local disks first, then upload them to object storage in background, this could speedup coping many small files into JuiceFS.

See ["Write Cache in Client"](guide/cache_management.md#writeback) for more information.
See ["Write Cache in Client"](guide/cache.md#writeback) for more information.

### Does JuiceFS support distributed cache?

Expand All @@ -104,7 +104,7 @@ See ["Write Cache in Client"](guide/cache_management.md#writeback) for more info

Yes, JuiceFS could be mounted using `juicefs` without root. The default directory for caching is `$HOME/.juicefs/cache` (macOS) or `/var/jfsCache` (Linux), you should change that to a directory which you have write permission.

See ["Read Cache in Client"](guide/cache_management.md#client-read-cache) for more information.
See ["Read Cache in Client"](guide/cache.md#client-read-cache) for more information.

## Access Related Questions

Expand Down
2 changes: 1 addition & 1 deletion docs/en/getting-started/for_distributed.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ JuiceFS guarantees a "close-to-open" consistency, which means that when two or m

#### Increase cache size to improve performance

Since object storage is a network-based storage service, it will inevitably encounter access latency. To solve this problem, JuiceFS provides and enables caching mechanism by default, i.e. allocating a part of local storage as a buffer layer between data and object storage, and caching data asynchronously to local storage when reading files. Please refer to ["Cache"](../guide/cache_management.md) for more details.
Since object storage is a network-based storage service, it will inevitably encounter access latency. To solve this problem, JuiceFS provides and enables caching mechanism by default, i.e. allocating a part of local storage as a buffer layer between data and object storage, and caching data asynchronously to local storage when reading files. Please refer to ["Cache"](../guide/cache.md) for more details.

JuiceFS will set 100GiB cache in `$HOME/.juicefs/cache` or `/var/jfsCache` directory by default. Setting a larger cache space on a faster SSD can effectively improve read and write performance of JuiceFS even more .

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
---
title: Cache
sidebar_position: 3
slug: /cache_management
---

For a file system driven by a combination of object storage and database, cache is an important medium for interacting efficiently between the local client and the remote service. Read and write data can be loaded into the cache in advance or asynchronously, and then the client uploads to or prefetches from the remote service in the background. The use of caching technology can significantly reduce the latency of storage operations and increase data throughput compared to interacting with remote services directly.
Expand Down
2 changes: 1 addition & 1 deletion docs/en/introduction/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ JuiceFS provides rich APIs for various forms of data management, analysis, archi
- **Distributed**: Each file system can be mounted on thousands of servers at the same time with high-performance concurrent reads and writes and shared data.
- **Strong Consistency**: Any changes committed to files are immediately visible on all servers.
- **Outstanding Performance**: JuiceFS achieves millisecond-level latency and nearly unlimited throughput depending on the object storage scale (see [performance test results](../benchmark/benchmark.md)).
- **Data Security**: JuiceFS supports encryption in transit and encryption at rest (view [Details](../security/encrypt.md)).
- **Data Security**: JuiceFS supports encryption in transit and encryption at rest (view [Details](../security/encryption.md)).
- **File Lock**: JuiceFS supports BSD lock (flock) and POSIX lock (fcntl).
- **Data Compression**: JuiceFS supports the [LZ4](https://lz4.github.io/lz4) and [Zstandard](https://facebook.github.io/zstd) compression algorithms to save storage space.

Expand Down
4 changes: 2 additions & 2 deletions docs/en/introduction/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Chunks exist to optimize lookup and positioning, while the actual file writing i

For example, if a file is generated through a continuous sequential write, each chunk contains only one slice. The figure above illustrates this scenario: a 160 MB file is sequentially written, resulting in three chunks, each containing only one slice.

File writing generates slices, and invoking `flush` persists these slices. `flush` can be explicitly called by the user, and even if not invoked, the JuiceFS client automatically performs `flush` at the appropriate time to prevent buffer overflow (refer to [buffer-size](../guide/cache_management.md#buffer-size)). When persisting to the object storage, slices are further split into individual *blocks* (default maximum size of 4 MB) to enable multi-threaded concurrent writes, thereby enhancing write performance. The previously mentioned chunks and slices are logical data structures, while blocks represent the final physical storage form and serve as the smallest storage unit for the object storage and disk cache.
File writing generates slices, and invoking `flush` persists these slices. `flush` can be explicitly called by the user, and even if not invoked, the JuiceFS client automatically performs `flush` at the appropriate time to prevent buffer overflow (refer to [buffer-size](../guide/cache.md#buffer-size)). When persisting to the object storage, slices are further split into individual *blocks* (default maximum size of 4 MB) to enable multi-threaded concurrent writes, thereby enhancing write performance. The previously mentioned chunks and slices are logical data structures, while blocks represent the final physical storage form and serve as the smallest storage unit for the object storage and disk cache.

![Split slices to blocks](../images/slice-to-block.svg)

Expand Down Expand Up @@ -68,5 +68,5 @@ However, it is not difficult to imagine that looking up the "most recently writt
Additional technical aspects of JuiceFS storage design:

* Irrespective of the file size, JuiceFS avoids storage merging to prevent read amplification and ensure optimal performance.
* JuiceFS provides strong consistency guarantees while allowing tuning options with caching mechanisms tailored to specific use cases. For example, by configuring more aggressive metadata caching, a certain level of consistency can be traded for enhanced performance. For more details, see [Metadata cache](../guide/cache_management.md#metadata-cache).
* JuiceFS provides strong consistency guarantees while allowing tuning options with caching mechanisms tailored to specific use cases. For example, by configuring more aggressive metadata caching, a certain level of consistency can be traded for enhanced performance. For more details, see [Metadata cache](../guide/cache.md#metadata-cache).
* JuiceFS supports the ["Trash"](../security/trash.md) functionality and enables it by default. After a file is deleted, it is retained for a certain period before being permanently cleared. This helps you avoid data loss caused by accidental deletion.
2 changes: 1 addition & 1 deletion docs/en/introduction/comparison/juicefs_vs_s3fs.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ S3FS does not limit the cache capacity by default, which may cause the cache to

JuiceFS uses a completely different caching approach than S3FS. First, JuiceFS guarantees data consistency. Secondly, JuiceFS defines a default disk cache usage limit of 100GiB, which can be freely adjusted by users as needed, and by default ensures that no more space is used when disk free space falls below 10%. When the cache usage limit reaches the upper limit, JuiceFS will automatically do cleanup using an LRU-like algorithm to ensure that cache is always available for subsequent read and write operations.

For more on JuiceFS caching, see [documentation](../../guide/cache_management.md).
For more on JuiceFS caching, see [documentation](../../guide/cache.md).

## Features

Expand Down
4 changes: 2 additions & 2 deletions docs/en/introduction/comparison/juicefs_vs_seaweedfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ SeaweedFS determines whether to compress data based on factors such as the file
Both support encryption, including encryption during transmission and at rest:

- SeaweedFS supports encryption both in transit and at rest. When data encryption is enabled, all data written to the volume server is encrypted using random keys. The corresponding key information is managed by the filer that maintains the file metadata. For details, see the [Wiki](https://github.com/seaweedfs/seaweedfs/wiki/Filer-Data-Encryption).
- For details about JuiceFS' encryption feature, see [Data Encryption](../../security/encrypt.md).
- For details about JuiceFS' encryption feature, see [Data Encryption](../../security/encryption.md).

## Client protocol comparison

Expand Down Expand Up @@ -177,7 +177,7 @@ Both support a CSI Driver. For details, see:

SeaweedFS client is equipped with [basic cache capabilities](https://github.com/seaweedfs/seaweedfs/wiki/FUSE-Mount), but its documentation weren't located at the time of writing, you can search for `cache` in the [source code](https://github.com/seaweedfs/seaweedfs/blob/master/weed/command/mount.go).

JuiceFS' client supports [metadata and data caching](../../guide/cache_management.md), allowing users to optimize based on their application's needs.
JuiceFS' client supports [metadata and data caching](../../guide/cache.md), allowing users to optimize based on their application's needs.

### Object storage gateway

Expand Down
Loading

0 comments on commit 3e02c31

Please sign in to comment.