v21 The Mercury Edition

This release has improvements across the board.

Top changes

1. Renamed the project to Gregory-AI. It's no longer just about multiple sclerosis
2. Better way to manage sources
3. Several improvements to email notifications and subscription lists
4. Add CTIS as a source for new clinical trials
5. Improves categories so that they are tied to only one research subject
6. Upgrade Django to 5.1.4
7. A LOT of clean up!

What's Changed

• Removes the field "source" because it is now a many-to-many relationship by @brunoamaral in #403
• Improvements to update_articles_info command by @brunoamaral in #405
• Update the query on the weekly digest to fetch articles using the new models.py by @brunoamaral in #407
• Hide deprecated fields 'relevant' and 'noun_phrases' by @brunoamaral in #408
• link categories to one or more subjects by @brunoamaral in #410
• fix wrong field type by @brunoamaral in #411
• dumb typO by @brunoamaral in #412
• Fixes python dependencies and migration errors by @brunoamaral in #416
• Do not use sudo when setting up the service by @mtpereira in #414
• removes setup tasks related to metabase and node-red by @brunoamaral in #417
• update setup instructions by @brunoamaral in #420
• fix: remove old source field from feedreader and add the new team and subject fields by @brunoamaral in #423
• renames "Gregory MS" to "GregoryAI" by @brunoamaral in #424
• adds missing query argument by @brunoamaral in #428
• add save method to make email addresses lowercase by @brunoamaral in #431
• add UniqueConstraint to email addresses by @brunoamaral in #432
• add filter options and active status of subscribers by @brunoamaral in #433
• Add options to subscribers admin by @brunoamaral in #434
• Add source id to import command by @brunoamaral in #435
• Improve email integration by @brunoamaral in #437
• Improves the way subscription lists work by @brunoamaral in #442
• require django 5.1.4 by @brunoamaral in #444
• Improvements to feedreader by @brunoamaral in #445
• Improve admin page for trials with filter options and search by @brunoamaral in #446
• Limit content of weekly digests and notifications to trials and articles found in the last 30 days by @brunoamaral in #447
• Improves the logic for assigning categories so that they are restricted to a research subject by @brunoamaral in #448
• make sure there is content before sending weekly digest by @brunoamaral in #449
• Adds CTIS as a source for clinical trials by @brunoamaral in #450
• adds a new 'active' boolean field to rss sources and updates feedreaders by @brunoamaral in #451
• Improve email notifications and add error logging by @brunoamaral in #452

New Contributors

• @mtpereira made their first contribution in #414

Full Changelog: v20...v21

New API endpoints to fetch content per team

What's Changed

• Add API endpoints to filter content by teams by @brunoamaral in #396
• Remove deprecated ML fields from the articles model by @brunoamaral in #399
• Add new API endpoints per team by @brunoamaral in #401

Full Changelog: v19...v20

v19 - Gregory Teams

We are going through some changes and will be breaking some things soon.

• Teams can be configured in the backend
• Each team can set their own categories, subjects, and sources
• Different subjects (topics) can use different Machine Learning Models

Right now some deprecated fields are still present in the database: relevant, categories, and ml_prediction_* for example.

Upgrading

Run manage.py makemigrations and manage.py migrate; then run manage.py migrate_categories.

What's Changed

• Send digest emails to the whole team in charge of that subject by @brunoamaral in #388
• removes metabase from docker compose and config file by @brunoamaral in #391
• Add support for multiple ml models per subject by @brunoamaral in #389
• Allow teams to configure their own categories by @brunoamaral in #393
• remove node-red from docker-compose by @brunoamaral in #394

Full Changelog: v18...v19

Multiple Research Subjects (topics)

1. We reviewed the pipeline that fetches and processes articles and clinical trials. it now runs with Django base commands: docker exec -it admin python manage.py
2. We added the option to create a Team using django-organizations
3. Each team can now have one or more research subjects and set sources to fetch articles for that subject.

Word of warning about the source field: the source field is no longer needed and was replaced with a ManyToMany field called sources. It is there for the time being, so that you can migrate your data using something like this:

from django.db import migrations

def copy_source_to_sources(apps, schema_editor):
    Article = apps.get_model('gregory', 'Articles')  # Replace 'gregory' with the actual app name
    for article in Article.objects.all():
        if article.source:  # Check if the old source field is not None
            article.sources.add(article.source)  # Add the old source to the new ManyToMany sources field

class Migration(migrations.Migration):

    dependencies = [
        ('gregory', '0077_articles_sources_alter_articles_source'),
    ]

    operations = [
        migrations.RunPython(copy_source_to_sources),
    ]

I suggest you create a team and subject and use the following to assign your articles to them:

def add_teams_and_subjects_to_articles():
	for article in Articles.objects.all():
		 article.teams.add(Team.objects.first())
		 article.subjects.add(Subject.objects.first())
	for trial in Trial.objects.all():
		 trial.teams.add(Team.objects.first())
		 trial.subjects.add(Subject.objects.first())

Right now, all the data is public and anyone can see what each team is researching. We want to make the API show a segregated view of the articles and clinical trials based on the user making the API request, with an option to set subjects as public or private. But I don't know how to do that.

Helping the Multiple Sclerosis Project

I have never asked for donations, but I am having a hard time keeping the MS project running. If you would like to help, you can donate to cover expenses through the Human Singularity Network, a non-profit we created to manage resources and partnerships.

https://donate.stripe.com/6oEeVmf1tdHIdOw7ss

Or if you can't, please share the project with everyone you feel might benefit from it.

Thank you!

What's Changed

• move db management command to the gregory app by @brunoamaral in #385
• New pipeline to process articles and clinical trials by @brunoamaral in #386
• Adding organisations to the user and article models by @brunoamaral in #387

Full Changelog: v17...v18

clean up release

I've lost the energy for witty texts.

We removed django-cron and updated the python packages, adding a new pipeline command that fetches and processes articles and clinical trials.

We also made it possible to import data from the WHO clinical trial registry.

What's Changed

• Improve who import by @brunoamaral in #364
• minor addition to the admin page by @brunoamaral in #365
• Add article list to authors' API endpoint by @brunoamaral in #366
• Improve feedreader to update trial if it exists by @brunoamaral in #367
• avoid error if api accesss expired. WIP by @brunoamaral in #363
• Add last updated date to trials by @brunoamaral in #368
• remove field for real_time_notification by @brunoamaral in #369
• Removes Django Cron and move all jobs to Django BaseCommands and adds track changes with django simple-history by @brunoamaral in #371
• Improve the workflow to import Articles and Trials and Records trial change history by @brunoamaral in #373
• upgrade packages by @brunoamaral in #377
• Create a single django command to run the full pipeline by @brunoamaral in #378
• update pipeline by @brunoamaral in #379
• Add error handling to orcid update, change the timeframe to query orcid by @brunoamaral in #380
• add missing import by @brunoamaral in #381
• remove redundant setting of variables by @brunoamaral in #382
• Update Documentation for Database Fields by @brunoamaral in #383
• add db diagram by @brunoamaral in #384

Full Changelog: v16...v17

v16

I was going to say this was mostly a maintenance release, but there are 3 big changes.

1. We have changed the license for Gregory. It is now under a Creative Commons — Attribution 4.0 International license.

This change was made because some people are working on amazing add-ons for Gregory and we want to do all we can to ensure their recognition.

2. Added fields for saving Machine Learning predictions made by the Linear SVC Support Vector Classifier.

3. Added a django command to import data from the World Health Organisation (WHO)

The Clinical Trials table was changed to include the same fields used by the WHO, so remember to run ./manage.py makemigrations && ./manage.py migrate.

After, you can now go to https://trialsearch.who.int/AdvSearch.aspx, find the clinical trials you want, and import them into gregory with ./manage.py importWHOXML FILE

What's Changed

• Update LICENSE to require attribution by @brunoamaral in #351
• 356 add to the trials table the same columns we have from the who ictrp by @brunoamaral in #357
• add identifier to value field to avoid ambiguity by @brunoamaral in #358
• 139 implement lsvc machine learning model by @brunoamaral in #361
• add lsvc prediction to api frontend by @brunoamaral in #362

Full Changelog: v15.5...v16

v15.5

Rest assured, this project is still very much alive. We have been doing a lot of backoffice work that we hope will give us more ways to develop Gregory.

For now, a quick maintenance release

What's Changed

• upgrade Django by @brunoamaral in #346
• add article count by @brunoamaral in #347
• Unique authors by @brunoamaral in #348
• Adds Country information to authors by @brunoamaral in #349
• deletes the twitter feed by @brunoamaral in #354
• Improve urls py by @brunoamaral in #355

Full Changelog: v15...v16

The Doc Brown Edition

Great Scott, it's been a while!

The team as been busy with other tasks and opening new doors. No fancy text this time, here are the changes since April.

1. Improvement to the sources, adding a list of fetch options instead of text field, with a new field for description of the source, just in case you need to keep notes.
2. A few api changes and endpoints to help build charts that we now see in the observatory page for Multiple Sclerosis research.
3. Check if a clinical trial is duplicate based on the ID number
4. Keep history of changes to a clinical trial, if possible, this needs more testing, if you can help.
5. We have included an authentication option using Django Rest Framework and djangorestframework-simplejwt

Remember to run manage.py makemigrations and manage.py migrate after pulling this new release

What's Changed

• For sources, change fetch method to a list of options by @brunoamaral in #327
• Add a Category slug by @brunoamaral in #330
• Adds category information to the API endpoints for Articles and Trials by @brunoamaral in #331
• Check if clinical trial is duplicate before saving, and update fields if needed. Also keep track of changes to clinical trials by @brunoamaral in #332
• Add get_monthly_article_counts and get_monthly_trial_counts by @brunoamaral in #333
• New api endpoint /categories/category_slug/monthly-counts/ by @brunoamaral in #334
• minor fix by @brunoamaral in #335
• include category name and slug in api response by @brunoamaral in #336
• add article count to categories by @brunoamaral in #338
• Authentication with jwt for the django rest framework by @brunoamaral in #341
• add missing files by @brunoamaral in #342
• 343 include an optional description field to the sources by @brunoamaral in #344

Full Changelog: v14...v15

The Virginia Lacy Jones Edition

Just because this one is short and sweet doesn't make it a small feet. We upgraded python under the hood, so rebuild your container when you're in an upgrading mood.

Size doesn't matter, quality does. We changed to a new summariser, much better than it was.

And for your favourite author, we now provide an RSS feed with all their articles. Find it at feed/articles/author/<author_id>/

What's Changed

• add timezone with dateutil.parse by @brunoamaral in #316
• upgrade python to v3.11 by @brunoamaral in #317
• avoids error if DOI is an empty string by @brunoamaral in #319
• Implement new summariser by @brunoamaral in #321
• Minor updates by @brunoamaral in #323
• cleaned up code by @brunoamaral in #322
• minor fixes in setup.py by @brunoamaral in #324
• adds an rss feed to every author, available at /feed/articles/author/<author_id>/ by @brunoamaral in #325

Full Changelog: v13.12...v14

The Bunny Edition

Not just bugs in this one, we've been working on trials. Now, we try to fetch the Eduract and NCT numbers for clinical trials, working our way to make sure we don't have duplicate content on that table. In the future, we want to move forward with tracking the progress of every clinical trial until completion.

What's Changed

• Add identifier numbers to table of clinical trials by @brunoamaral in #304
• add source object when saving clinical trials from rss feed by @brunoamaral in #306
• Add identifiers to public API by @brunoamaral in #307
• implements ClinicalTrial class and moves code to functions.py by @brunoamaral in #313
• create a clinicaltrial class for processing new information by @brunoamaral in #310
• 291 docker compose fails to build django container by @brunoamaral in #292
• add a new api endpoint to view relevant articles in the given week by @brunoamaral in #294
• some fixes for the newsletter api by @brunoamaral in #295
• fixes queries in the newsletter api by @brunoamaral in #296
• fixes the error in running makemigrations on a fresh install by @brunoamaral in #299
• upgrade packages by @brunoamaral in #303
• 311 move remove utm to functionspy by @brunoamaral in #312

Full Changelog: v13...v14