Species config UI (#375)

* feat: add species settings management interface

* feat: add Species link to sidebar navigation

* refactor: update species settings structure and improve filtering logic

- Renamed 'customThresholds' to 'Thresholds' for consistency.
- Enhanced the addConfig and removeConfig methods to work with the new Thresholds structure.
- Replaced the filterSpecies method with updatePredictions for better clarity and functionality.
- Updated input bindings in the HTML to reflect the new method names and data structure.
- Adjusted hidden input fields to submit the updated species settings correctly.

* refactor: update settings handling for species thresholds

- Removed the 'realtime.species.threshold' field from the fieldsToSkip map for improved configuration management.
- Added special handling for SpeciesThreshold in the updateStructFromForm function to support JSON unmarshaling of species thresholds.
- Enhanced map handling for species thresholds, allowing for dynamic updates from form values.
- Improved logging for better debugging during struct updates.

* refactor: restructure species settings for improved clarity and functionality

- Updated SpeciesSettings to include a new structure for species thresholds, consolidating confidence thresholds and action configurations.
- Renamed fields for consistency and clarity, enhancing the overall configuration management.
- Introduced SpeciesThreshold and SpeciesAction types to better represent per-species configurations and actions.
- Improved documentation for better understanding of the species-related settings.

* refactor: streamline species configuration handling and improve action processing

- Removed the deprecated species configuration loading logic, consolidating species settings into a new structure for better clarity and maintainability.
- Updated action processing to utilize the new species thresholds and action configurations, enhancing the filtering logic for species inclusion/exclusion.
- Introduced helper functions for better management of script parameters and action handling, ensuring a more modular approach.
- Improved logging for debugging purposes, providing clearer insights into species configuration usage during processing.

* refactor: update actions modal in species settings for improved functionality

- Changed the action retrieval logic to source from species thresholds, enhancing the accuracy of action configurations.
- Updated the modal structure to utilize a more consistent styling approach with modal classes.
- Improved input field classes for better UI consistency and user experience.
- Added a backdrop to the modal for improved usability when closing the modal.

* feat: enhance species settings UI with script execution capabilities

- Added a new input field for specifying the script path, allowing users to provide the full path to the script they want to execute.
- Updated the parameters section to display available parameters for the script, enabling users to easily add them via buttons.
- Introduced a clear parameters button to reset the selected parameters, improving user experience.
- Disabled the action type selection to prevent changes while the script execution feature is being configured.

* refactor: update species settings input handling and prediction logic

- Replaced 'allSpecies' with 'allowedSpecies' to filter species based on prepared settings.
- Simplified the 'updatePredictions' method to utilize 'allowedSpecies' for generating predictions.
- Updated input fields to use a unified datalist for species suggestions, enhancing user experience.
- Removed unnecessary parameters from the prediction update function, streamlining the code.

* feat: add safeJSON function for secure JSON handling in templates

- Introduced a new function, safeJSONFunc, to safely convert values to escaped JSON strings for use in HTML templates.
- Enhanced security by ensuring proper escaping of JSON strings to prevent XSS vulnerabilities.
- Updated the GetTemplateFunctions method to include the new safeJSON function, improving template rendering capabilities.

* feat: enhance settings content rendering with threshold handling and improved error logging

- Added special handling for species thresholds in the renderSettingsContent method, converting thresholds into a formatted string for better display.
- Improved error logging by providing a detailed template data dump when rendering fails, aiding in debugging and issue resolution.

* restore removed comment

* feat: add debug logging functionality to BirdNET

- Introduced a new Debug method in the BirdNET struct to enable conditional logging based on the debug mode setting.
- Enhanced the logging capabilities to provide detailed output for debugging purposes, improving the overall traceability of the application.

* feat: enhance range filtering logic in GetProbableSpecies

- Added debug logging to track the application of the range filter and species exclusion.
- Implemented checks to exclude species based on a configurable exclude list.
- Introduced functions to process included species and species with configured actions, ensuring maximum scores are assigned correctly.
- Improved species matching logic to handle both common and scientific names, enhancing filtering accuracy.

* feat: add Debug method for conditional logging in handlers

- Introduced a new Debug method in the Handlers struct to enable debug logging based on the web server's debug mode setting.
- Enhanced logging capabilities to provide detailed output when debug mode is enabled, improving traceability and debugging efficiency.

* feat: refactor species action configuration from thresholds to a unified config structure

- Renamed ExecuteScriptAction to ExecuteCommandAction and updated its implementation to execute commands instead of scripts.
- Changed species action configuration from using thresholds to a new config structure, allowing for more flexible action definitions.
- Updated related components to reflect the new configuration structure, including UI adjustments in species settings for command execution.
- Enhanced the handling of species actions in the processor and rendering logic to accommodate the new configuration format.

* fix: update species action configuration to include command field

- Modified the species action configuration to include a 'command' field in addition to 'parameters', enhancing the structure for executing commands.
- Updated the logic in the actions modal to reflect the new command field, ensuring proper handling of species actions.
- Adjusted the push logic for actions to accommodate the new command structure, improving the overall functionality of species settings.

* refactor: streamline species filtering and threshold retrieval logic

- Removed redundant include/exclude checks from the processResults method, simplifying the filtering process for species.
- Updated the getBaseConfidenceThreshold method to utilize a new configuration structure for species thresholds, enhancing clarity and maintainability.
- Improved debug logging to reflect changes in threshold handling, ensuring better traceability during species processing.

* feat: add security debug flag to default configuration

- Introduced a new default setting for 'security.debug' to enable or disable detailed logging in the security configuration.

* feat: enhance species action management in settings

- Introduced a new action index to track the current action being edited.
- Updated the actions modal to retrieve and set existing actions or default values.
- Refactored action saving logic to replace the existing action instead of pushing to an array.
- Improved button interactions to prevent default behavior and enhance user experience.
- Added functionality to clear parameters and improved parameter handling in the UI.

* fix: round confidence score to two decimal places in observation creation

* refactor: rename and enhance parameter parsing in action processing

* feat: enhance command execution with validation and sanitization

- Added command path validation to ensure commands are absolute and executable.
- Implemented safe argument building with parameter name validation and sanitization.
- Improved logging for command execution, including success and error outputs.
- Introduced helper functions for environment management and value sanitization.

* feat: enhance settings handling with improved logging and data validation

- Added detailed logging for debugging during settings updates and JSON unmarshaling.
- Implemented cleanup of Actions data to ensure Parameters are properly initialized.
- Enhanced error handling with more informative log messages for better traceability.

* feat: improve logging for species settings rendering

- Added debug logging to output the species configuration being passed to the template for better traceability.
- Removed unnecessary threshold handling code to streamline the rendering process.

* fix: format threshold values and enforce numeric input in species settings

- Updated the display of threshold values to round to two decimal places for better readability.
- Changed the input model for new thresholds to enforce numeric input, ensuring only valid numbers are accepted.

* feat: integrate control channel for config reload management

- Updated the RealtimeAnalysis function to utilize the new control channel in the processor and HTTP server initialization.
- Enhanced the Processor and Server constructors to accept the control channel, enabling coordinated reload control across components.
- Implemented a control signal monitor in the Processor for managing config reload requests.

* feat: implement control signal handling and range filter reloading

- Added a new control signal monitor in the Processor to handle reload requests for the range filter.
- Introduced a ReloadRangeFilter function to update the species list based on the current date.
- Enhanced the Handlers to include a control channel for sending reload signals when relevant settings change.
- Implemented a check for changes in range filter settings to trigger reloads, improving configuration management.

* feat: add debug mode to range filter configuration

- Introduced a new debug flag in the RangeFilterSettings struct to enable debug mode.
- Updated default configuration to include the debug setting for the range filter, defaulting to false.
- Ensured that the range filter model and threshold settings remain intact while enhancing configuration options.

* refactor: rename reload control signal and update handling logic

- Renamed the control signal from ReloadRangeFilter to RebuildRangeFilter for clarity.
- Updated the control signal monitor to call BuildRangeFilter instead of ReloadRangeFilter.
- Enhanced logging messages to reflect the changes in functionality, indicating a rebuild instead of a reload.

* fix: reintroduce included species list check which was accidentally removed

* feat: rename and enhance range filter function with debug logging

- Renamed ReloadRangeFilter to BuildRangeFilter for improved clarity.
- Added detailed logging to track species and scores during the filter build process.
- Introduced a debug mode that writes included species to a file when enabled, enhancing traceability for debugging purposes.

* enh: add AllSpecies list which contains all labels and use it for included species list predictions

* feat: add GetIncludedSpecies and GetAllSpecies functions for species management

- Introduced GetIncludedSpecies to return a deduplicated list of included species, sorting both common and scientific names.
- Added GetAllSpecies to provide a deduplicated list of all available species from the handlers, ensuring unique entries and sorting.
- Removed the prepareSpeciesData function from utils.go to streamline species data preparation, consolidating functionality within the Server struct.

* feat: add GetLabels function to retrieve available species labels

- Introduced GetLabels method in Handlers to return a list of all species labels from BirdNET settings, enhancing species management capabilities.

* refactor: update label handling to use Settings structure

- Replaced direct usage of Labels field in BirdNET with Labels from Settings.BirdNET across multiple files.
- Ensured consistent label management by centralizing label storage within the Settings structure, improving maintainability and clarity.
- Updated relevant functions to reflect this change, enhancing the overall architecture of the BirdNET module.

* refactor: remove redundant species data preparation from renderSettingsContent

* refactor: clean up debug logging in settings handler

- Removed commented-out debug logging statements in SaveSettings and updateStructFromForm functions to improve code readability.
- Renamed parameters in rangeFilterSettingsChanged function for clarity, enhancing maintainability.
- Ensured consistent handling of settings changes without unnecessary debug output, streamlining the settings management process.

* refactor: rename control signal for range filter rebuild

* feat: enhance species settings management with include/exclude functionality

- Added sections for including and excluding species in the settings interface, allowing users to specify species that should always be included or excluded from detection.
- Implemented dynamic handling of species lists with the ability to add and remove species, as well as configure custom actions and thresholds for specific species.
- Improved user experience with clear labeling and descriptions for each section, ensuring intuitive interaction with species settings.
- Introduced real-time updates to the settings state, enhancing responsiveness and usability.

* refactor: simplify species settings management by removing unused variables and methods

- Removed redundant variables and methods related to species configuration and action handling, streamlining the species settings interface.
- Consolidated prediction updates to utilize a single species list for both inclusion and exclusion, enhancing code clarity and maintainability.
- Improved the initialization of species settings to ensure default values are correctly set, preventing potential errors.
- Updated input handling for species addition to improve user experience and reduce complexity in the codebase.

* refactor: streamline Processor initialization and species filtering logic

- Removed the control channel parameter from the Processor's New function, simplifying its initialization.
- Consolidated species filtering logic to utilize a single range filter species list, enhancing clarity and maintainability.
- Cleaned up the processResults function to improve readability and ensure consistent handling of species detection.

* refactor: enhance species settings interface with improved structure and functionality

* refactor: update species settings title for clarity

- Changed the title for the species settings route from "Editor" to "Species Settings" to better reflect its purpose and improve user understanding.

* refactor: improve actions modal in species settings interface

- Enhanced the actions modal by adding keyboard accessibility with an escape key listener for closing the modal.
- Updated the modal's maximum height and overflow behavior to improve usability on smaller screens.
- Adjusted the backdrop styling for better visibility and user experience when the modal is open.

* refactor: streamline BirdNET initialization and control signal handling

- Removed the initializeIncludedSpecies function, consolidating its logic into the new BuildRangeFilter function within the birdnet package.
- Updated the RealtimeAnalysis function to utilize a buffered control channel and removed unnecessary parameters from the processor initialization.
- Introduced a new startControlMonitor function to handle control signals for range filter rebuilding, enhancing modularity and clarity.
- Deleted obsolete control signal handling code from the processor package, simplifying the overall architecture.

* style: enhance scrollbar and overflow behavior for improved user experience

- Added stable scrollbar gutter to ensure consistent layout across browsers.
- Implemented auto-scrolling for html and body to enhance page navigation.
- Adjusted drawer content to allow scrolling with a minimum height of 100vh.
- Overrode DaisyUI overflow constraints for better drawer functionality.

assets/custom.css

@@ -1,3 +1,28 @@
+/* Ensure scrollbar gutter is stable */
+html {
+  scrollbar-gutter: stable !important;
+}
+
+/* Force page scrolling behavior */
+html, body {
+  height: 100%;
+  overflow-y: auto;
+}
+
+/* Ensure drawer content can scroll */
+.drawer-content {
+  height: auto !important;
+  min-height: 100vh;
+  overflow-y: auto;
+  scrollbar-gutter: stable;
+}
+
+/* Override any DaisyUI overflow constraints */
+.drawer.drawer-open {
+  overflow-y: auto;
+  scrollbar-gutter: stable;
+}
+
 .audio-control {
   width: 100%;
   height: 25px;

internal/analysis/birdnet_init.go

@@ -2,7 +2,6 @@ package analysis
 
 import (
 	"fmt"
-	"time"
 
 	"github.com/tphakala/birdnet-go/internal/birdnet"
 	"github.com/tphakala/birdnet-go/internal/conf"
@@ -21,31 +20,9 @@ func initializeBirdNET(settings *conf.Settings) error {
 		}
 
 		// Initialize included species list
-		if err := initializeIncludedSpecies(settings); err != nil {
-			return fmt.Errorf("failed to initialize included species: %w", err)
+		if err := birdnet.BuildRangeFilter(bn); err != nil {
+			return fmt.Errorf("failed to initialize BirdNET: %w", err)
 		}
 	}
 	return nil
 }
-
-// initializeIncludedSpecies initializes the included species list based on date and geographic location
-func initializeIncludedSpecies(settings *conf.Settings) error {
-	speciesScores, err := bn.GetProbableSpecies(time.Now(), 0.0)
-	if err != nil {
-		return fmt.Errorf("error getting probable species: %w", err)
-	}
-
-	// Update included species in settings
-	var includedSpecies []string
-	for _, speciesScore := range speciesScores {
-		includedSpecies = append(includedSpecies, speciesScore.Label)
-	}
-	settings.UpdateIncludedSpecies(includedSpecies)
-
-	// debug print included species in human readable format
-	/*for _, species := range includedSpecies {
-		fmt.Printf("Included species: %s\n", species)
-	}*/
-
-	return nil
-}

internal/analysis/processor/execute.go

@@ -3,50 +3,168 @@ package processor
 
 import (
 	"fmt"
+	"log"
+	"os"
 	"os/exec"
+	"path/filepath"
 	"reflect"
+	"runtime"
+	"strings"
 
 	"github.com/tphakala/birdnet-go/internal/datastore"
 )
 
-type ExecuteScriptAction struct {
-	ScriptPath string
-	Params     map[string]interface{}
+type ExecuteCommandAction struct {
+	Command string
+	Params  map[string]interface{}
 }
 
-// A map to store the action configurations for different species
-//var speciesActionsMap map[string]SpeciesActionConfig
+func (a ExecuteCommandAction) Execute(data interface{}) error {
+	log.Printf("[analysis/processor/execute] Executing command: %s params: %v\n", a.Command, a.Params)
 
-func (a ExecuteScriptAction) Execute(data interface{}) error {
-	//log.Println("Executing script:", a.ScriptPath)
 	// Type assertion to check if data is of type Detections
 	detection, ok := data.(Detections)
 	if !ok {
 		return fmt.Errorf("ExecuteScriptAction requires Detections type, got %T", data)
 	}
 
-	// Building the command line arguments from the Params map
+	// Validate and resolve the command path
+	cmdPath, err := validateCommandPath(a.Command)
+	if err != nil {
+		return fmt.Errorf("invalid command path: %w", err)
+	}
+
+	// Building the command line arguments with validation
+	args, err := buildSafeArguments(a.Params, detection.Note)
+	if err != nil {
+		return fmt.Errorf("error building arguments: %w", err)
+	}
+
+	log.Printf("[analysis/processor/execute] Command: %s, Args: %v\n", cmdPath, args)
+
+	// Create command with validated path and arguments
+	cmd := exec.Command(cmdPath, args...)
+
+	// Set a clean environment
+	cmd.Env = getCleanEnvironment()
+
+	// Execute the command
+	output, err := cmd.CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("error executing command: %w, output: %s", err, string(output))
+	}
+
+	log.Printf("[analysis/processor/execute] Command executed successfully: %s", string(output))
+	return nil
+}
+
+// validateCommandPath ensures the command exists and is executable
+func validateCommandPath(command string) (string, error) {
+	// Clean the path to remove any potential directory traversal
+	command = filepath.Clean(command)
+
+	// Check if it's an absolute path
+	if !filepath.IsAbs(command) {
+		return "", fmt.Errorf("command must use absolute path: %s", command)
+	}
+
+	// Verify the file exists and is executable
+	info, err := os.Stat(command)
+	if err != nil {
+		return "", fmt.Errorf("command not found: %w", err)
+	}
+
+	// Check file permissions
+	if runtime.GOOS != "windows" {
+		if info.Mode()&0111 == 0 {
+			return "", fmt.Errorf("command is not executable: %s", command)
+		}
+	}
+
+	return command, nil
+}
+
+// buildSafeArguments creates a sanitized list of command arguments
+func buildSafeArguments(params map[string]interface{}, note datastore.Note) ([]string, error) {
 	var args []string
-	for key, value := range a.Params {
-		// Fetching the value from detection.Note using reflection
-		noteValue := getNoteValueByName(detection.Note, key)
+
+	for key, value := range params {
+		// Validate parameter name (allow only alphanumeric and _-)
+		if !isValidParamName(key) {
+			return nil, fmt.Errorf("invalid parameter name: %s", key)
+		}
+
+		// Get value from Note or use default
+		noteValue := getNoteValueByName(note, key)
 		if noteValue == nil {
-			noteValue = value // Use default value if not found in Note
+			noteValue = value
 		}
 
-		arg := fmt.Sprintf("--%s=%v", key, noteValue)
+		// Convert and validate the value
+		strValue, err := sanitizeValue(noteValue)
+		if err != nil {
+			return nil, fmt.Errorf("invalid value for parameter %s: %w", key, err)
+		}
+
+		// Quote the value if it contains spaces or special characters
+		if strings.ContainsAny(strValue, " \t\n\r\"'") {
+			// Escape any existing quotes in the value
+			strValue = strings.ReplaceAll(strValue, `"`, `\"`)
+			// Wrap the value in quotes
+			strValue = fmt.Sprintf(`"%s"`, strValue)
+		}
+
+		arg := fmt.Sprintf("--%s=%s", key, strValue)
 		args = append(args, arg)
 	}
 
-	// Executing the script with the provided arguments
-	cmd := exec.Command(a.ScriptPath, args...)
-	output, err := cmd.CombinedOutput()
-	if err != nil {
-		return fmt.Errorf("error executing script: %w, output: %s", err, string(output))
+	return args, nil
+}
+
+// isValidParamName checks if a parameter name contains only safe characters
+func isValidParamName(name string) bool {
+	for _, r := range name {
+		if !((r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') ||
+			(r >= '0' && r <= '9') || r == '_' || r == '-') {
+			return false
+		}
 	}
+	return true
+}
 
-	//fmt.Printf("Script executed successfully: %s", string(output))
-	return nil
+// sanitizeValue converts and validates a value to string
+func sanitizeValue(value interface{}) (string, error) {
+	// Convert to string and validate
+	str := fmt.Sprintf("%v", value)
+
+	// Basic sanitization - remove any control characters
+	str = strings.Map(func(r rune) rune {
+		if r < 32 {
+			return -1
+		}
+		return r
+	}, str)
+
+	// Additional validation can be added here
+
+	return str, nil
+}
+
+// getCleanEnvironment returns a minimal set of necessary environment variables
+func getCleanEnvironment() []string {
+	// Provide only necessary environment variables
+	env := []string{
+		"PATH=" + os.Getenv("PATH"),
+		"TEMP=" + os.Getenv("TEMP"),
+		"TMP=" + os.Getenv("TMP"),
+	}
+
+	// Add system root for Windows
+	if runtime.GOOS == "windows" {
+		env = append(env, "SystemRoot="+os.Getenv("SystemRoot"))
+	}
+
+	return env
 }
 
 func getNoteValueByName(note datastore.Note, paramName string) interface{} {

internal/analysis/processor/processor.go

@@ -41,6 +41,7 @@ type Processor struct {
 	pendingMutex        sync.Mutex // Mutex to protect access to pendingDetections
 	lastDogDetectionLog map[string]time.Time
 	dogDetectionMutex   sync.Mutex
+	controlChan         chan string
 }
 
 // DynamicThreshold represents the dynamic threshold configuration for a species.
@@ -99,9 +100,6 @@ func New(settings *conf.Settings, ds datastore.Interface, bn *birdnet.BirdNET, m
 	// Start the held detection flusher
 	p.pendingDetectionsFlusher()
 
-	// Load Species configs
-	p.Settings.Realtime.Species, _ = LoadSpeciesConfig(conf.SpeciesConfigCSV)
-
 	// Initialize BirdWeather client if enabled in settings
 	if settings.Realtime.Birdweather.Enabled {
 		var err error
@@ -270,12 +268,8 @@ func (p *Processor) processResults(item queue.Results) []Detections {
 			continue
 		}
 
-		// Match against location-based filter
-		//if !p.Settings.IsSpeciesIncluded(result.Species) {
-		if !p.Settings.IsSpeciesIncluded(scientificName) {
-			if p.Settings.Debug {
-				log.Printf("Species not on included list: %s\n", commonName)
-			}
+		// Match species against range filter included species list
+		if !contains(p.Settings.BirdNET.RangeFilter.Species, speciesLowercase) {
 			continue
 		}
 
@@ -327,13 +321,16 @@ func (p *Processor) handleHumanDetection(item queue.Results, speciesLowercase st
 
 // getBaseConfidenceThreshold retrieves the confidence threshold for a species, using custom or global thresholds.
 func (p *Processor) getBaseConfidenceThreshold(speciesLowercase string) float32 {
-	confidenceThreshold, exists := p.Settings.Realtime.Species.Threshold[speciesLowercase]
-	if !exists {
-		confidenceThreshold = float32(p.Settings.BirdNET.Threshold)
-	} else if p.Settings.Debug {
-		log.Printf("\nUsing confidence threshold of %.2f for %s\n", confidenceThreshold, speciesLowercase)
+	// Check if species has a custom threshold in the new structure
+	if config, exists := p.Settings.Realtime.Species.Config[speciesLowercase]; exists {
+		if p.Settings.Debug {
+			log.Printf("\nUsing custom confidence threshold of %.2f for %s\n", config.Threshold, speciesLowercase)
+		}
+		return float32(config.Threshold)
 	}
-	return confidenceThreshold
+
+	// Fall back to global threshold
+	return float32(p.Settings.BirdNET.Threshold)
 }
 
 // generateClipName generates a clip name for the given scientific name and confidence.
@@ -474,3 +471,14 @@ func (p *Processor) pendingDetectionsFlusher() {
 		}
 	}()
 }
+
+// Helper function to check if a slice contains a string (case-insensitive)
+func contains(slice []string, item string) bool {
+	item = strings.ToLower(item)
+	for _, s := range slice {
+		if strings.ToLower(s) == item {
+			return true
+		}
+	}
+	return false
+}