Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate contact's metadata into native columns #1670

Merged
merged 7 commits into from
Jan 8, 2025
Merged

Migrate contact's metadata into native columns #1670

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ee6bb4d
Migrate contact's metadata into native columns
samnang Dec 18, 2024
2d5e6d0
Update data migration
samnang Dec 19, 2024
34879c6
Update data migration
samnang Jan 7, 2025
8f4c159
Update data migrations
samnang Jan 7, 2025
48a58fa
Merge branch 'develop' into contact-data-migrations
samnang Jan 7, 2025
a549c32
Merge branch 'develop' into contact-data-migrations
samnang Jan 8, 2025
4c91377
Update data migration for ZM
samnang Jan 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions lib/tasks/data_migrations/migrate_contact_metadata_to_native_fields.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
namespace :data_migrations do
task migrate_contact_metadata_to_native_fields: :environment do
Account.find_each do |account|
puts "Migrating contacts for account: #{account.id}"

account.contacts.find_each do |contact|
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

account.contacts.where(iso_country_code: nil) This will allow us to run it multiple times in case of an error

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. We can say if iso_country_code has the value, we already migrated or they created from the new codes.

ApplicationRecord.transaction do
contact.iso_country_code = PhonyRails.country_from_number(contact.msisdn)
contact.language_code = contact.metadata["language_code"]
contact.save!

# EWS Cambodia
contact.metadata.fetch("commune_ids", []).each do | commune_id |
commune = Pumi::Commune.find_by_id(phone_call_metadata(:commune_code))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line might return nil. I think there will be some invalid communes in there

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, it's was not assigned by the user, but still there are some invalid codes? We will add a check to skip it.


contact.addresses.find_or_create_by!(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a unique index on [beneficiary_id, iso_region_code, administrative_division_level_2_code, administrative_division_level_3_code, administrative_division_level_4_code]?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also thought about that too, if we add the index, we should also at least add not null on iso_region_code.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After I thought on it again, we shouldn't add or reply on this because some of those columns can be null.

It is possible a contact has iso_region_code, administrative_division_level_2_code, administrative_division_level_3_name instead, so create_or_find_by may not work.

iso_region_code: commune.province.iso3166_2,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember we discussed about denormalizing iso_country_code in the addresses table, but I can't remember did we decide to do it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we discussed and just leave them for now, but I couldn't remember 100% as well 😂

administrative_division_level_2_code: commune.district_id,
administrative_division_level_3_code: commune.id
)
end

# EWS Laos
contact.metadata.fetch("registered_districts", []).each do | district_code |
district = CallFlowLogic::EWSLaosRegistration::DISTRICTS.find { |d| d.code == district_code }

contact.addresses.find_or_create_by!(
iso_region_code: district.province.iso3166,
samnang marked this conversation as resolved.
Show resolved Hide resolved
administrative_division_level_2_code: district.code,
)
end

# Africa's Voices (Somalia)
# metadata: {"name"=>"John Doe", "district"=>"Kalabo", "language"=>"Lozi", "province"=>"Western"}

# PIN Zambia
# metadata: {"name"=>"John Doe", "address"=>"Newa", "district"=>"Nalolo", "facility"=>"Mouyo", "language"=>"silozi", "province"=>"Western", "date_of_registration"=>"29/10/2021"}
Copy link
Collaborator

@dwilkie dwilkie Dec 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

language_code = "loz" #  # https://en.wikipedia.org/wiki/Lozi_language
iso_region_code = "ZM-01" # https://en.wikipedia.org/wiki/ISO_3166-2:ZM
administrative_division_level_2_name = "Nalolo"
administrative_division_level_3_name = "Mouyo"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For language_code, currently we don't have any validations.

For iso_region_code, I will pull it from there to lookup the name into the code.

end
end
end
end
end
Loading