Skip to content

Latest commit

 

History

History
69 lines (51 loc) · 4.56 KB

README.md

File metadata and controls

69 lines (51 loc) · 4.56 KB

Back to Database Requirements

Data Integration

The integration of data from partners is critical to the success of the Sports Hub Portal, as it provides up-to-date news and information related to sports. The ETL (extract, transform, load) process should involve retrieving news data from partners on a regular basis, performing data cleaning and validation, and migrating the data to the Operational DB every hour. To achieve this, the system must meet both functional and non-functional requirements described below.

1 Functional Requirements

  1. The system must retrieve news data from partners every 30 minutes, extract the relevant information, and store it in the Raw DB.
  2. The system must be able to handle different data formats and APIs.
  3. The system must ensure that only relevant data is extracted and stored in the Raw DB.
  4. The system must be able to handle API errors and retry failed requests.
  5. The system must perform data cleaning and transformation as necessary before storing the data in the Raw DB.
  6. The system must remove duplicates and ensure that data is consistent and accurate.
  7. The system must conform to predefined data quality standards.
  8. The system must validate incoming data against predefined business rules to ensure data consistency and accuracy.
  9. If the news data contains a date field, the system must ensure that the date is within a valid range.
  10. If the data contains text fields, the system must ensure that they contain only valid characters and are within a valid length range.
  11. Text fields must not contain invalid characters, such as symbols or emojis.
  12. The system must ensure that there are no null values in the required fields.
  13. The system must ensure that data types are consistent and match the schema of the Raw DB.
  14. The system must migrate data from the Raw DB to the Operational DB every hour.
  15. The system must transform the data to conform to the Operational DB schema.
  16. The system must ensure that only relevant data is migrated to the Operational DB.
  17. The system must be able to handle large volumes of data and maintain system performance.
  18. The system must be able to handle high concurrency and scale as necessary.
  19. The system must minimize system downtime during data migration.
  20. The system must provide a user interface for monitoring the status of the ETL process and identifying errors or anomalies in the data.
  21. The system must be able to send alerts to designated personnel when errors or anomalies are detected.

2 Non-Functional Requirements

Performance

  1. The system must be able to handle large volumes of data and maintain system performance during data retrieval, cleaning, and migration processes.
  2. The system must be implemented using parameters and variables to facilitate efficient data pipeline execution.
  3. Each pipeline execution must be completed in a timely manner to avoid overlapping with the next execution.
  4. The system must be optimized to minimize processing time and resource consumption.

Scalability

  1. The system must be able to scale horizontally or vertically as necessary to accommodate future growth in data volume or user traffic.
  2. The system must be designed to support distributed processing and load balancing.
  3. The system must be able to handle increases in data volume without affecting performance or availability.
  4. The system must be able to support the addition of new data sources or APIs without significant impact on the system.

Security

  1. The system must ensure that data is secure and protected from unauthorized access or modification.
  2. The system must employ encryption for sensitive data during transmission and storage.
  3. The system must authenticate and authorize users to access data based on predefined roles and permissions.

Reliability

  1. The system must be reliable and maintain data integrity during the ETL process.
  2. The system must ensure that data is not lost or corrupted during the ETL process.
  3. The system must provide a failover mechanism in the event of system failures or disruptions.

Maintainability

  1. The system must be easy to maintain and modify as necessary.
  2. The system must use modular and maintainable code to facilitate future modifications or upgrades.

NOTE: Recommended resources for partner’s data integration: https://mediastack.com/_