Add POC disk cleanup #117
Add POC disk cleanup #117
Conversation
WalkthroughThe recent update introduces a disk cleanup mechanism within the analysis processor. This addition is designed to monitor disk usage and automatically clear out files once certain thresholds are reached. It's a smart move towards ensuring the system remains efficient and avoids running out of storage space unexpectedly. The update encompasses both the implementation of the cleanup functionality and its testing. Changes
Recent Review StatusConfiguration used: CodeRabbit UI Files ignored due to path filters (2)
Files selected for processing (2)
Additional Context UsedGitHub Check Runs (1)
Additional comments not posted (6)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
|
Sorry.... Was going to push a pr to my own fork such that I did not forget about the branch... |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
|
||
| import ( | ||
| "fmt" | ||
| "io/ioutil" |
There was a problem hiding this comment.
Replace deprecated ioutil package with os or io package.
- import "io/ioutil"
+ import "os"And update ioutil.ReadDir(filePath) to os.ReadDir(filePath) in the countFiles function.
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| "io/ioutil" | |
| import "os" |
| } | ||
| } | ||
|
|
||
| // TODO add a test that expects an error if too many files have to be removed at once. |
There was a problem hiding this comment.
Add missing test case for error handling.
It's important to cover scenarios where errors are expected, such as when too many files need to be removed. Would you like assistance in implementing this test case?
| func (a DatabaseAction) DiskCleanUp(calculator DiskUsageCalculator) error { | ||
| basePath := conf.GetBasePath(a.Settings.Realtime.AudioExport.Path) | ||
|
|
||
| const cleanupThreshold = 0.9 // Example threshold for disk cleanup (90% usage) | ||
|
|
||
| // Calculate bytesToRemove based on available disk space and cleanup threshold | ||
| diskUsed, err := calculator.Used(basePath) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| diskSize, err := calculator.Size(basePath) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| bytesToRemove := int64(diskUsed) - int64(float64(diskSize)*cleanupThreshold) | ||
|
|
||
| // Only cleanup if required (positive bytes to remove) | ||
| if bytesToRemove < 0 { | ||
| return nil | ||
| } | ||
|
|
||
| log.Printf("Reached disk usage treshold of %f percent. Will try to remove: %d bytes of files.\n", cleanupThreshold*100, bytesToRemove) | ||
|
|
||
| var files []os.FileInfo | ||
| var totalSize int64 | ||
|
|
||
| errr := filepath.Walk(basePath, | ||
| func(path string, info os.FileInfo, err error) error { | ||
| if err != nil { | ||
| return err | ||
| } | ||
| // Check if the file is a regular file | ||
| if info.Mode().IsRegular() { | ||
| files = append(files, info) | ||
| totalSize += info.Size() // Accumulate file sizes | ||
| } | ||
| return nil | ||
| }) | ||
| if errr != nil { | ||
| log.Println(errr) | ||
| return err | ||
| } | ||
|
|
||
| // Sort files by modification time | ||
| sort.Sort(ByModTime(files)) | ||
|
|
||
| // Check if it's possible to remove X bytes | ||
| if totalSize < bytesToRemove { | ||
| return fmt.Errorf("failed to remove %d bytes: insufficient files to meet the target size", bytesToRemove) | ||
| } | ||
|
|
||
| // Remove files until total size exceeds or equals bytesToRemove | ||
| var removedSize int64 | ||
| for i := 0; i < len(files) && removedSize < bytesToRemove; i++ { | ||
| err := os.Remove(filepath.Join(basePath, files[i].Name())) | ||
| if err != nil { | ||
| log.Printf("Error removing file %s: %v\n", files[i].Name(), err) | ||
| } else { | ||
| removedSize += files[i].Size() // Update removedSize after removal | ||
| log.Printf("File %s (%d bytes) removed successfully. %d bytes remaning to remove.\n", files[i].Name(), files[i].Size(), max(bytesToRemove-removedSize, 0)) | ||
| } | ||
| } | ||
|
|
||
| // Perform a check afterward again. It could be the case that because of removal errors, | ||
| // it is still not possible to remove enough files to reach the threshold | ||
| if removedSize < bytesToRemove { | ||
| return fmt.Errorf("only managed to remove %d bytes: insufficient files to meet the target size %d", removedSize, bytesToRemove) | ||
| } | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
Refine disk cleanup logic for better error handling and efficiency.
The disk cleanup logic in DiskCleanUp method is comprehensive but can be refined for better error handling and efficiency. Consider the following improvements:
- Use
filepath.WalkDirinstead offilepath.Walkfor better performance in Go 1.16 and later. - Directly return the error from
filepath.WalkDirinstead of logging and returning a different error. - Accumulate errors during file removal to attempt removing as many files as possible before returning an error.
| @@ -105,6 +209,9 @@ | |||
| if a.Settings.Realtime.AudioExport.Enabled { | |||
| time.Sleep(1 * time.Second) // Sleep for 1 second to allow the audio buffer to fill | |||
| pcmData, _ := a.AudioBuffer.ReadSegment(a.Note.BeginTime, time.Now()) | |||
|
|
|||
| a.DiskCleanUp(RealDiskUsage{}) | |||
There was a problem hiding this comment.
Check error return value of DiskCleanUp.
- a.DiskCleanUp(RealDiskUsage{})
+ if err := a.DiskCleanUp(RealDiskUsage{}); err != nil {
+ log.Printf("Disk cleanup failed: %v", err)
+ }It's important to handle the error returned by DiskCleanUp to ensure that any issues during the cleanup process are properly logged or addressed.
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| a.DiskCleanUp(RealDiskUsage{}) | |
| if err := a.DiskCleanUp(RealDiskUsage{}); err != nil { | |
| log.Printf("Disk cleanup failed: %v", err) | |
| } |
No description provided.