My iOS Core Data Heavyweight Migration Failure Story

My Failure Story for You

2

Let me show you what to do when you're painting and things aren't quite going the way you want or expect.
You just paint over it like this. Doing this gives birth to a completely beautiful new tree. We don't make mistakes. We just have happy accidents!
- Bob Ross

Hello, I'm Leedi. 🧑🏻‍💻

My Failure for You

At DelightRoom, we have a culture called 'My Failure for You'. It means that by learning lessons through failure and sharing them, we can save others from paying the same cost. Today, I want to save you some time by sharing a failure of my own.

Recently, we've been running a massive update on Alarmy's DB structure.
And my initial plan for this task looked like this:

1. Merge the DB, which was split between Realm and Core Data, into Core Data.

2. Migrate the Core Data data to V3.

There was a lot of trial and error during this process, especially in step 2, and thanks to that, I was able to learn some great lessons. Today, I'd like to share what I learned along the way.

First of all, here is why I chose Core Data:

1. Trust in a First-Party Framework.

2. Ease of transitioning to SwiftData later on.

I had high hopes that once we moved everything to Core Data this time, migrating to SwiftData later would be smooth sailing.

Core Data Migration

First, as you probably all know, Core Data's migration methods are broadly divided into the automatic approach known as Lightweight migration, and the manual approach known as Heavyweight migration.

Generally, we use the Lightweight method. That is, for light structural changes like adding a row, or renaming an entity or property.

According to developer.apple.com, the Lightweight method is discussed for cases like the following:

- Adding an attribute
- Removing an attribute
- A non-optional attribute becomes optional
- An optional attribute becomes non-optional and defines a default value
- Renaming an entity or attribute

Here's an example.

If you rename an entity from ALColorEntity—which you thought would be used universally—to ALCarColorEntity because it’s now only used for cars, or if you change the property name of an RGB value inside an entity to rgb, Lightweight is plenty sufficient. (Since you can easily find resources on Lightweight migration by Googling, I'll skip further explanation.)

However, our DB doesn't just need this kind of light migration. Sometimes we need to add default values to new columns, or compute and change them into different values during the migration.

In this case, we need to migrate using the Heavyweight method. If you look up Heavyweight migration methods, you'll find a few ways to do it, but personally, I liked this tutorial the best.

Instead of those unfriendly Apple docs, a kind raywenderlich tutorial really puts the mind at ease.

To summarize, it looks like this:

1. Add a new model and change the Model Version.
2. Create a MappingModel with the Source and Destination configured.
3. Create an object that defines the detailed rules and adopt NSEntityMigrationPolicy.
4. And then register this object in the CustomPolicy.

Then, the Heavyweight migration will execute at the app's launch time.

Super easy, right?

But we aren't just going to do this kind of migration once.
V1 → V2 → V3, as time goes by, the versions will pile up. And this is exactly where all the problems begin...

Realm Migration

For a while now, Alarmy has been managing most of its DB using Realm. We were only managing the recently added sleep-related data via Core Data.

If we look at how migration works in Realm, it goes like this:

1. Register the new schemaVersion in Realm.Configuration.
2. Put the detailed migration logic inside the migrationBlock.
3. Pass the Configuration in before initializing Realm.
4. Realm runs the migrationBlock and performs the migration.

Inside the migrationBlock, we can easily run the migration using the oldSchemaVersion and the Migration object utilized for the process.

In Realm, it doesn't matter if the migration schema updates from v1 → v2 → v3. As long as we write the code to run the migrations sequentially, we only have to handle the transition from the immediately preceding model!

Core Data Migrating multiple versions

So, how does Core Data handle migrations across multiple versions like this?

Basically, Core Data detects a mismatch between the model used in the store and the current model in the bundle. If they don't match, it automatically attempts to migrate. Because of this, Core Data doesn't support sequential migrations version-by-version.

In other words, if a user with a V1 model and a user with a V2 model both update to the latest V3, it operates like this:

Honestly, it took quite a long time just to figure out this behavior.
Because I naturally assumed it would remember the versions sequentially.

Getting back on track, the problem this causes is obvious. As the Version goes up, the maintenance cost increases exponentially. If Version 4 comes out, you need 3 mappers: [V1 → V4, V2 → V4, V3 → V4]. So for version N, you end up needing N-1 Mapping Models.

Let's imagine time goes by and we make it to V10.

We'd have to figure out a migration from the ancient, entirely forgotten V1 all the way to V10. Just thinking about it is awful..

Maybe this is Apple's grand design to get us to minimize migrations...
But I really hate that design...

Of course, maybe the gods at Apple knew about this problem too, because at WWDC 2023, they gave us a way to solve these Core Data Migration issues through a Staged approach.

https://developer.apple.com/videos/play/wwdc2023/10186

So incredibly appealing!

Except for the fact that...
it only works starting from iOS 17.... *seethes*..

Core Data progressive migration

But it's too early to despair. There's no way our senior engineers have just been suffering through this problem helplessly.

Searching for a solution using the keyword 'Core Data progressive migration', I found a few super useful articles.

PROGRESSIVE CORE DATA MIGRATIONS
by.williamboles

Core Data Progressive Migrations
by.kean

And I even found a Framework applying this concept.

GitHub — prisma-ai/Sworm: CoreData based Swift ORM

(🫢 Oh wow... I knew you wouldn't let me down..)

Looking through the content, they all perform sequential migrations in a similar way.

Take your time reading the blogs for the detailed explanations. To summarize the core idea further:

1. Manually register the order of the versions.
2. Use the NSManagedObjectModel API to identify the current version of the SourceModel.
3. Follow the MigrationSteps sequentially.
4. Migration complete 🥳

Oh! So easy. I tested it out, and I can confirm it works perfectly.

And with that happy ending, I wrap up this migration post..
Is what I wanted to say, but Core Data is not that easy to deal with..

Applying it to a live product is a whole different story..

Now, the problem occurred in the most unexpected place.

When attempting a migration to the current version from before a specific commit, a situation occurred where the current version of the SourceModel couldn't be identified.

Even when I opened up the metadata, all the values were right there, so I couldn't figure out the root cause.
Forcing the version just caused a crash during the mapping process...
Not knowing the internal logic of the configuration really killed my sense of stability.

Phew..!
Seriously.. I suddenly started to understand why people use Realm..

It’s a bit of an abrupt conclusion to this post, but
with that final attempt, my journey to conquer Core Data migration came to an end.

And it wasn't just because of the reasons mentioned above either.

1. Lingering anxiety about stability
   - My understanding of Core Data was still low, and even if I solved the issue above, the anxiety over stability wouldn't seem to go away.

2. The cost was getting way too high
   - Our goal was to deliver the desired value to the user safely and quickly. But we judged that if we kept working this way, it would be hard to achieve either safety or speed.

3. Unable to modify the Model and MappingModel through code, making changes hard to track.
   - Just like why we don't use Storyboards, you can't check changes via code here, and we judged that letting this pile up would harm our stability.

After deep-diving into Core Data to a certain extent, what we actually needed became perfectly clear.

I ultimately decided to migrate all the Core Data models over to Realm. Thankfully, since we were accessing the Repository through an interface, the cost of making that change wasn't too massive.

And just like that, our app is currently going through QA in a much more stable state.

Wrapping Up

To sum it up, I took away two main lessons this time around.

First, I was able to level up my understanding of Core Data's migration logic.

• Since I've mostly built apps where server communication was the priority, I've never really had the chance to play around with a Local DB like this before. Through this opportunity, I deepened my understanding of how Core Data and Realm operate, and thanks to that, I gained confidence in our future direction.

Second, even if it's a First-Party Framework, you still have to verify it to check its stability.

• If we had just trusted the POC results and shipped the app, users on versions before a certain point would have experienced crashes, which could have been fatal for our app. In the end, they are all just tools, and I once again felt the absolute necessity of verification.

But that doesn't mean I've given up on First-Party Frameworks.

After shaking this off completely, polishing up, and deepening our understanding, the plan is to eventually bounce back from Realm to SwiftData next time. Of course, even in that case, we will have to verify its safety and usefulness through various experiments.

Thanks for reading to the end!