test: dependency updates (#130)

* refactor: use admin.register decorator * build: pin django-mptt dependency * fix: update verbose_name_plural in questionset * fix rdmorganiser#791 for openid_connect providers and refactor keycloak snippet * Add custom {more} markdown tag (rdmorganiser#595) * Adjust details/summary style * Add ValueConflictValidator and check for conflicts when storing values * Autosave after eraseValue and removeValue * Add tests for ValueConflictValidator * Fix human2bytes utils function * chore: refactor human2bytes utils func * Fix ValueQuotaValidator and add tests * Update HUMAN2BYTES_MAPPER and add test * Refactor progress bar and take sets and conditions into account (rdmorganiser#596) * Check is_optional for progress bar (again) * Add progress_count and progress_total to Project model, compute on POST when using the API and add progress to the project overview (rdmorganiser#488) * Add navigation action to ProjectViewSet and refactor navigation (rdmorganiser#299, rdmorganiser#424) * Always show count in progress bar * Fix progress bar for questionsets * Fix progress computation * Fix progress permissions * Add tests for progress and navigation * Add progress to project hierarchy (rdmorganiser#488) * chore: refactor values query to managers * Use "count of total" for the progress in projects table * Add support for BASE_URL to the management react interface (rdmorganiser#815) * Rename variable * Refresh navigation after auto-save in interview * chore: add pytest-playwright dependency * chore: add custom pytest marker for e2e tests * chore: enable debug mode per env, default false * refactor: add "fixtures" fixture to reuse in e2e tests * test: add frontend-tests with pytest-playwright Co-authored-by: David Wallace Fixes: rdmorganiser#715 * ci: run e2e tests in ci * ci: add dependency groups to dependabot config * chore: update ruff repo name in pre-commit config * build: update pre-commit hooks * chore: fix ruff errors (PGH002, UP032) --------- Co-authored-by: David Wallace <[email protected]> Co-authored-by: Jochen Klar <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
afuetterer · Nov 23, 2023 · 7114d06 · 7114d06
1 parent a54867e
commit 7114d06
Show file tree

Hide file tree

Showing 71 changed files with 1,357 additions and 321 deletions.
diff --git a/.eslintrc.js b/.eslintrc.js
@@ -44,5 +44,5 @@ module.exports = {
     'react': {
       'version': 'detect'
     }
-  }
+  },
 }
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -1,32 +1,77 @@
 version: 2
 updates:
-  - package-ecosystem: pip
-    directory: /
-    schedule:
-      interval: monthly
-    open-pull-requests-limit: 10
-    target-branch: dependency-updates
-    labels:
-      - dependencies
-      - python
-      - type:maintenance
-  - package-ecosystem: github-actions
-    directory: /
-    schedule:
-      interval: monthly
-    open-pull-requests-limit: 10
-    target-branch: dependency-updates
-    labels:
-      - dependencies
-      - github_actions
-      - type:maintenance
-  - package-ecosystem: npm
-    directory: /
-    schedule:
-      interval: monthly
-    open-pull-requests-limit: 10
-    target-branch: dependency-updates
-    labels:
-      - dependencies
-      - javascript
-      - type:maintenance
+- package-ecosystem: pip
+  directory: /
+  schedule:
+    interval: monthly
+  open-pull-requests-limit: 10
+  target-branch: dependency-updates
+  labels:
+  - dependencies
+  - python
+  - type:maintenance
+  ignore:
+  - dependency-name: django-mptt # pinned, 0.15 requires Python >= 3.9
+  groups:
+    # create a single pull request containing all updates for the optional dependencies
+    optional:
+      patterns:
+      - coveralls
+      - django-allauth
+      - django-auth-ldap
+      - gunicorn
+      - mysqlclient
+      - pre-commit
+      - psycopg*
+      - pytest*
+    # create a single pull request containing all updates for django related dependencies
+    django:
+      patterns:
+      - django*
+      - drf*
+- package-ecosystem: github-actions
+  directory: /
+  schedule:
+    interval: monthly
+  open-pull-requests-limit: 10
+  target-branch: dependency-updates
+  labels:
+  - dependencies
+  - github_actions
+  - type:maintenance
+  groups:
+    # create a single pull request containing all updates for GitHub Actions
+    github-actions:
+      patterns:
+      - '*'
+- package-ecosystem: npm
+  directory: /
+  schedule:
+    interval: monthly
+  open-pull-requests-limit: 10
+  target-branch: dependency-updates
+  labels:
+  - dependencies
+  - javascript
+  - type:maintenance
+  groups:
+    react:
+      patterns:
+      - react*
+    redux:
+      patterns:
+      - redux*
+    babel:
+      patterns:
+      - '@babel*'
+      - babel*
+    webpack:
+      patterns:
+      - webpack*
+    eslint:
+      patterns:
+      - eslint*
+    prod-dependencies:
+      dependency-type: production
+    dev-dependencies:
+      dependency-type: development
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -92,6 +92,30 @@ jobs:
         GITHUB_DB_BACKEND: ${{ matrix.db-backend }}
         COVERALLS_FLAG_NAME: '${{ matrix.db-backend }}: ${{ matrix.python-version }}'
         COVERALLS_PARALLEL: true
+    # end-to-end tests
+    - uses: actions/setup-node@v4
+      with:
+        node-version: 18
+        cache: npm
+      if: matrix.python-version == '3.12' && matrix.db-backend == 'postgres'
+    - name: Cache Playwright browsers
+      uses: actions/cache@v3
+      with:
+        path: ~/.cache/ms-playwright/
+        key: playwright-browsers
+      if: matrix.python-version == '3.12' && matrix.db-backend == 'postgres'
+    - name: Install e2e tests dependencies
+      run: |
+        npm install
+        npm run build:prod
+        playwright install chromium
+      if: matrix.python-version == '3.12' && matrix.db-backend == 'postgres'
+    - name: Run end-to-end tests
+      run: pytest -p randomly -p no:cacheprovider --reuse-db --numprocesses=auto --dist=loadscope -m e2e --nomigrations
+      if: matrix.python-version == '3.12' && matrix.db-backend == 'postgres'
+      env:
+        DJANGO_DEBUG: True
+        GITHUB_DB_BACKEND: ${{ matrix.db-backend }}
 
   coveralls:
     name: Indicate completion to coveralls
@@ -140,20 +164,6 @@ jobs:
       - run: python -m pip freeze
       - run: python -m pip list --outdated
 
-  webpack-build:
-    name: Test webpack-build
-    needs: lint
-    runs-on: ubuntu-22.04
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v3
-        with:
-          node-version: 18
-          cache: npm
-      - run: npm install --dev
-      - run: npm run build
-      - run: npm run build:prod
-
   required-checks-pass:
     if: always()
     needs:
@@ -162,7 +172,6 @@ jobs:
       - coveralls
       - dev-setup
       - optional-dependencies
-      - webpack-build
     runs-on: ubuntu-22.04
     steps:
       - uses: re-actors/alls-green@release/v1

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -9,7 +9,7 @@ repos:
     hooks:
       - id: check-hooks-apply
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v4.5.0
     hooks:
       - id: check-ast
       - id: check-json
@@ -21,13 +21,13 @@ repos:
       - id: trailing-whitespace
         exclude: \.dot$
       - id: debug-statements
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: v0.0.291
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.1.6
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
   - repo: https://github.com/pre-commit/mirrors-eslint
-    rev: v8.50.0
+    rev: v8.54.0
     hooks:
       - id: eslint
         args: [--fix, --color]

diff --git a/conftest.py b/conftest.py
@@ -10,29 +10,33 @@
 from rdmo.accounts.utils import set_group_permissions
 
 
+@pytest.fixture(scope="session")
+def fixtures():
+    allowed_file_stems = {
+        'accounts',
+        'conditions',
+        'domain',
+        'groups',
+        'options',
+        'overlays',
+        'projects',
+        'questions',
+        'sites',
+        'tasks',
+        'users',
+        'views'
+    }
+    fixtures = []
+    for fixture_dir in settings.FIXTURE_DIRS:
+        filenames = [filename for filename in Path(fixture_dir).iterdir() if filename.stem in allowed_file_stems]
+        fixtures.extend(filenames)
+    return fixtures
+
+
 @pytest.fixture(scope='session')
-def django_db_setup(django_db_setup, django_db_blocker):
+def django_db_setup(django_db_setup, django_db_blocker, fixtures):
     """Populate database with test data from fixtures directories."""
     with django_db_blocker.unblock():
-        fixtures = []
-        for fixture_dir in settings.FIXTURE_DIRS:
-            for file in Path(fixture_dir).iterdir():
-                if file.stem in [
-                    'accounts',
-                    'conditions',
-                    'domain',
-                    'groups',
-                    'options',
-                    'overlays',
-                    'projects',
-                    'questions',
-                    'sites',
-                    'tasks',
-                    'users',
-                    'views'
-                ]:
-                    fixtures.append(file)
-
         call_command('loaddata', *fixtures)
         set_group_permissions()
 
@@ -49,7 +53,4 @@ def files(settings, tmp_path):
 @pytest.fixture
 def json_data():
     json_file = Path(settings.BASE_DIR) / 'import' / 'catalogs.json'
-    json_data = {
-        'elements': json.loads(json_file.read_text())
-    }
-    return json_data
+    return {'elements': json.loads(json_file.read_text())}
diff --git a/pyproject.toml b/pyproject.toml
@@ -37,6 +37,9 @@ dynamic = [
   "version",
 ]
 dependencies = [
+  # dependencies with major version on zero are declared with
+  # major.minor.patch, because they can potentially introduce breaking changes
+  # in minor version updates anytime
   "defusedcsv~=2.0",
   "defusedxml~=0.7.1",
   "django~=4.2",
@@ -46,7 +49,7 @@ dependencies = [
   "django-filter~=23.2",
   "django-libsass~=0.9",
   "django-mathfilters~=1.0",
-  "django-mptt~=0.14.0",
+  "django-mptt==0.14.0", # pinned, 0.15 requires Python >= 3.9
   "django-rest-swagger~=2.2",
   "django-settings-export~=1.2",
   "django-split-settings~=1.2",
@@ -89,6 +92,7 @@ pytest = [
   "pytest-cov~=4.1",
   "pytest-django~=4.5",
   "pytest-mock~=3.11",
+  "pytest-playwright~=0.4.3",
   "pytest-randomly~=3.15",
   "pytest-xdist~=3.3",
 ]
@@ -177,7 +181,10 @@ DJANGO_SETTINGS_MODULE = "config.settings"
 testpaths = ["rdmo"]
 python_files = "test_*[!.txt].py"
 pythonpath = [".", "testing"]
-addopts = "-p no:randomly"
+addopts = '-p no:randomly -m "not e2e"'
+markers = [
+  "e2e: marks tests as end-to-end tests using playwright (deselect with '-m \"not e2e\"')",
+]
 filterwarnings = [
   # fail on RemovedInDjango50Warning exception
   "error::django.utils.deprecation.RemovedInDjango50Warning",

diff --git a/rdmo/accounts/admin.py b/rdmo/accounts/admin.py
@@ -5,21 +5,25 @@
 from .models import AdditionalField, AdditionalFieldValue, ConsentFieldValue, Role
 
 
+@admin.register(AdditionalField)
 class AdditionalFieldAdmin(admin.ModelAdmin):
     pass
 
 
+@admin.register(AdditionalFieldValue)
 class AdditionalFieldValueAdmin(admin.ModelAdmin):
     readonly_fields = ('user', )
 
 
+@admin.register(ConsentFieldValue)
 class ConsentFieldValueAdmin(admin.ModelAdmin):
     readonly_fields = ('user', 'consent')
 
     def has_add_permission(self, request, obj=None):
         return False
 
 
+@admin.register(Role)
 class RoleAdmin(admin.ModelAdmin):
     search_fields = ('user__username', 'user__email')
     list_filter = ('member', 'manager', 'editor', 'reviewer')
@@ -50,9 +54,3 @@ def editors(self, obj):
 
     def reviewers(self, obj):
         return self.render_all_sites_or_join(obj, 'reviewer')
-
-
-admin.site.register(AdditionalField, AdditionalFieldAdmin)
-admin.site.register(AdditionalFieldValue, AdditionalFieldValueAdmin)
-admin.site.register(ConsentFieldValue, ConsentFieldValueAdmin)
-admin.site.register(Role, RoleAdmin)
diff --git a/rdmo/accounts/templates/socialaccount/snippets/provider_list.html b/rdmo/accounts/templates/socialaccount/snippets/provider_list.html
@@ -24,13 +24,23 @@
         <img src="{% static 'accounts/img/orcid-signin.png' %}" alt="ORCID sign in" />
     </a>
 </li>
-{% elif provider.id == 'keycloak' %}
-<li class="socialaccount_provider_break">
-    <a title="{{ provider.name }}" class="socialaccount_provider {{ provider.id }}"
-        href="{% provider_login_url provider.id process=process scope=scope auth_params=auth_params %}">
-        <img src="{% static 'accounts/img/keycloak_logo_200px.svg' %}" alt="Keycloak sign in" />
-    </a>
-</li>
+{% elif provider.id == 'openid_connect' %}
+    {% if provider.app.provider_id == 'keycloak' %}
+    <li class="socialaccount_provider_break">
+        <a title="{{ provider.name }}" class="socialaccount_provider {{ provider.id }}"
+            href="{% provider_login_url provider process=process scope=scope auth_params=auth_params %}">
+            <img src="{% static 'accounts/img/keycloak_logo_200px.svg' %}" alt="Keycloak sign in" />
+        </a>
+    </li>
+    {% else %}
+    <li>
+        <a title="{{provider.name}}" class="socialaccount_provider {{provider.id}}"
+            href="{% provider_login_url provider process=process scope=scope auth_params=auth_params %}">
+            {{ provider.name }}
+        </a>
+    </li>
+{% endif %}
+
 {% else %}
 <li>
     <a title="{{provider.name}}" class="socialaccount_provider {{provider.id}}"

diff --git a/rdmo/conditions/admin.py b/rdmo/conditions/admin.py
@@ -17,6 +17,7 @@ def clean(self):
         ConditionLockedValidator(self.instance)(self.cleaned_data)
 
 
+@admin.register(Condition)
 class ConditionAdmin(admin.ModelAdmin):
     form = ConditionAdminForm
 
@@ -25,6 +26,3 @@ class ConditionAdmin(admin.ModelAdmin):
     readonly_fields = ('uri', )
     list_filter = ('relation', )
     filter_horizontal = ('editors', )
-
-
-admin.site.register(Condition, ConditionAdmin)
-Original file line number
+Diff line change
@@ Expand Up / @@ -44,5 +44,5 @@ module.exports = { @@
         'react': {
           'version': 'detect'
         }
-      }
+      },
     }