In our app under development we are using Core Data with a sqlite backing store to store our data. The object model for our app is complex. Also, the total amount of data served by our app is too large to fit into an iOS (iPhone/iPad/iPod Touch) app bundle. Because of the fact that our users are, typically, interested only in a subset of the data, we've partitioned our data in such a way that the app ships with a subset (albeit, ~100 MB) of the data objects in the app bundle. Our users have the option of downloading additional data objects (of size ~5 MB to 100 MB) from our server after they pay for the additional contents through iTunes in-app purchases. The incremental data files (existing in sqlite backing stores) use the same xcdatamodel version as the data that ships with the bundle; there is zero changes to the object model. The incremental data files are downloaded from our server as a gzipped sqlite files. We don't want to bloat our app bundle by shipping the incremental contents with the app. Also, we don't want to rely on queries over webservice (because of the complex data model). We've tested the download of the incremental sqlite data from our server. We have been able to add the downloaded data store to the app's shared persistentStoreCoordinator.
在我们正在开发的应用程序中,我们使用Core Data和sqlite后备存储来存储我们的数据。我们的应用程序的对象模型很复杂。此外,我们的应用程序提供的数据总量太大,无法容纳iOS(iPhone / iPad / iPod Touch)应用程序包。由于我们的用户通常只对数据的一个子集感兴趣,因此我们对数据进行了分区,使得应用程序附带了一个子集(尽管大约100 MB)的数据对象。应用包。我们的用户可以选择在通过iTunes应用内购买支付额外内容后,从我们的服务器下载其他数据对象(大小约为5 MB到100 MB)。增量数据文件(存在于sqlite后备存储中)使用与bundle附带的数据相同的xcdatamodel版本;对象模型没有任何变化。增量数据文件作为gzip sqlite文件从我们的服务器下载。我们不希望通过随应用程序发送增量内容来扩大我们的应用程序包。此外,我们不希望依赖于webservice上的查询(因为复杂的数据模型)。我们测试了从服务器上下载的增量sqlite数据。我们已经能够将下载的数据存储添加到应用程序的共享persistentStoreCoordinator中。
{
NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:defaultStoreURL options:options error:&error])
{
NSLog(@"Failed with error: %@", [error localizedDescription]);
abort();
}
// Check for the existence of incrementalStore
// Add incrementalStore
if (incrementalStoreExists) {
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:incrementalStoreURL options:options error:&error])
{
NSLog(@"Add of incrementalStore failed with error: %@", [error localizedDescription]);
abort();
}
}
}
However, there are two problems with doing it this way.
但是,这样做有两个问题。
- Data fetch results (e.g., with NSFetchResultController) show up with the data from the incrementalStoreURL appended to the end of the data from the defaultStoreURL.
- Some of the objects are duplicated. There are many entities with read-only data in our data model; these get duplicated when we add the second persistentStore to the persistentStoreCoordinator.
数据获取结果(例如,使用NSFetchResultController)显示来自incrementalStoreURL的数据附加到defaultStoreURL的数据末尾。
一些对象是重复的。在我们的数据模型中有许多具有只读数据的实体;当我们将第二个persistentStore添加到persistentStoreCoordinator时,这些会被复制。
Ideally, we would like Core Data to merge the object graphs from the two persistent stores into one (there are no shared relationships between data from the two stores at the time of the data download). Also, we would like to remove the duplicate objects. Searching the web, we saw a couple of questions by people attempting to do the same thing we are doing--such as this answer and this answer. We've read Marcus Zarra's blog on importing large data sets in Core Data. However, none of the solutions we've seen worked for us. We don't want to manually read and save the data from the incremental store to the default store as we think this will be very slow and error prone on the phone. Is there a more efficient way of doing the merge?
理想情况下,我们希望Core Data将两个持久存储中的对象图合并为一个(数据下载时两个存储中的数据之间没有共享关系)。此外,我们想删除重复的对象。在网上搜索时,我们看到了一些人试图做同样事情的问题 - 比如这个答案和这个答案。我们已经阅读了Marcus Zarra关于在Core Data中导入大型数据集的博客。但是,我们看到的解决方案都没有为我们工作。我们不希望手动读取增量存储中的数据并将数据保存到默认存储中,因为我们认为这将非常慢并且在手机上容易出错。有没有更有效的合并方式?
We've attempted to solve the problem by implementing a manual migration as follows. However, we haven't been able to successfully get the merge to happen. We are not really clear on the solution suggested by answers 1 and 2 referenced above. Marcus Zarra's blog addressed some of the issues we had at the outset of our project importing our large dataset into iOS.
我们尝试通过实现手动迁移来解决问题,如下所示。但是,我们无法成功实现合并。我们对上面提到的答案1和2提出的解决方案并不十分清楚。 Marcus Zarra的博客解决了我们在项目开始时将大型数据集导入iOS的一些问题。
{
NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
NSMigrationManager *migrator = [[NSMigrationManager alloc] initWithSourceModel:__managedObjectModel destinationModel:__managedObjectModel];
if (![migrator migrateStoreFromURL:stateStoreURL
type:NSSQLiteStoreType
options:options
withMappingModel:nil
toDestinationURL:destinationStoreURL
destinationType:NSSQLiteStoreType
destinationOptions:nil
error:&error])
{
NSLog(@"%@", [error userInfo]);
abort();
}
}
It seems that the author of answer 1 ended up reading his data from the incremental store and saving to the default store. Perhaps, we've misunderstood the solution suggested by both articles 1 & 2. The size of our data may preclude us from manually reading and re-inserting our incremental data into the default store. My question is: what is the most efficient way to get the object graphs from two persistentStores (that have the same objectModel) to merge into one persistentStore?
似乎答案1的作者最终从增量存储中读取他的数据并保存到默认存储。也许,我们误解了第1条和第2条所建议的解决方案。我们的数据大小可能会阻止我们手动读取和重新插入我们的增量数据到默认存储中。我的问题是:从两个持久存储(具有相同的objectModel)中获取对象图的最有效方法是什么才能合并到一个persistentStore中?
Automatic migration works pretty well when we add new entity attributes to object graphs or modify relationships. Is there a simple solution to merging similar data into the same persistent store that will be resilient enough to stop and resume--as automatic migration is done?
当我们向对象图添加新的实体属性或修改关系时,自动迁移非常有效。是否有一个简单的解决方案将类似的数据合并到同一个持久存储中,该存储具有足够的弹性以便停止和恢复 - 因为自动迁移已完成?
3 个解决方案
#1
6
After several attempts, I've figured out how to make this work. The secret is to first create the incremental store data without any data for the read-only entities. Without leaving read-only data out of the incremental stores, the entities instances for these would get duplicated after the data migration and merge. Hence, the incremental stores should be created without these read-only entities. The default store will be the only store that has them.
经过几次尝试,我已经想出了如何使这项工作。秘诀是首先创建增量存储数据,而不包含只读实体的任何数据。如果不从增量存储中删除只读数据,则在数据迁移和合并之后,这些实体的实体实例将会重复。因此,应该在没有这些只读实体的情况下创建增量存储。默认商店将是唯一拥有它们的商店。
For example, I had entities "Country" and "State" in my data model. I needed to have only one instance of Country and State in my object graph. I kept these entities out of incremental stores and created them only in the default store. I used Fetched Properties to loosely link my main object graph to these entities. I created the default store with all the entity instances in my model. The incremental stores either didn't have the read-only entities (i.e., Country and State in my case) to start with or deleted them after data creation is completed.
例如,我的数据模型中有实体“Country”和“State”。我需要在对象图中只有一个Country和State实例。我将这些实体保留在增量存储之外,并仅在默认存储中创建它们。我使用Fetched Properties将我的主对象图松散地链接到这些实体。我在模型中创建了包含所有实体实例的默认存储。增量存储在数据创建完成后没有只读实体(即我的国家和国家)开始或删除它们。
Next step is to add the incremental store to it's own persistentStoreCoordinator (not the same as the coordinator for the default store that we want to migrate all contents to) during application startup.
下一步是在应用程序启动期间将增量存储添加到它自己的persistentStoreCoordinator(与我们要将所有内容迁移到的默认存储的协调器不同)。
The final step is to call migratePersistentStore method on the incremental store to merge its data to the main (i.e., default) store. Presto!
最后一步是在增量存储上调用migratePersistentStore方法,将其数据合并到主(即默认)存储中。普雷斯托!
The following code fragment illustrates the last two steps I mentioned above. I did these steps to make my setup to merge incremental data into a main data store to work.
以下代码片段说明了我上面提到的最后两个步骤。我做了这些步骤,使我的设置将增量数据合并到主数据存储中。
{
NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:defaultStoreURL options:options error:&error])
{
NSLog(@"Failed with error: %@", [error localizedDescription]);
abort();
}
// Check for the existence of incrementalStore
// Add incrementalStore
if (incrementalStoreExists) {
NSPersistentStore *incrementalStore = [_incrementalPersistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:incrementalStoreURL options:options error:&error];
if (!incrementalStore)
{
NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
abort();
}
if (![_incrementalPersistentStoreCoordinator migratePersistentStore:incrementalStore
toURL:_defaultStoreURL
options:options
withType:NSSQLiteStoreType
error:&error])
{
NSLog(@"%@", [error userInfo]);
abort();
}
// Destroy the store and store coordinator for the incremental store
[_incrementalPersistentStoreCoordinator removePersistentStore:incrementalStore error:&error];
incrementalPersistentStoreCoordinator = nil;
// Should probably delete the URL from file system as well
//
}
}
#2
1
The reason your migration isn't working is because the managed object model is identical.
迁移不起作用的原因是托管对象模型相同。
Technically, you're talking about "data migration" not "schema migration". CoreData's migration API is designed for schema migration, that is handling changes to the managed object model.
从技术上讲,您所说的是“数据迁移”而不是“模式迁移”。 CoreData的迁移API专为模式迁移而设计,即处理对托管对象模型的更改。
As far as transferring data from one store to another you're kind of on your own. CoreData can help you be efficient by using batching and fetch limits on your fetch requests, but you need to implement the logic yourself.
至于将数据从一个商店转移到另一个商店,你就是自己的一种。 CoreData可以通过对获取请求使用批处理和获取限制来帮助您提高效率,但您需要自己实现逻辑。
It sounds like you have two persistent stores, a big one and a small one. It would be most efficient to load the small one and analyze it, discovering the set of primary keys or unique identifiers you need to query for in the larger store.
听起来你有两个持久性商店,一个大商店和一个小商店。加载小的并分析它,发现在较大的商店中需要查询的主键集或唯一标识符是最有效的。
You could then de-dupe easily by simply querying the larger store for those identifiers.
然后,您可以通过简单地查询较大的商店来获取这些标识符来轻松删除。
The documentation for NSFetchRequest has the API for scoping your queries:
NSFetchRequest的文档具有用于确定查询范围的API:
#3
1
You don't need any migration - migration is designed to bring changes in NSManagedObjectModel,not in data itself.
您不需要任何迁移 - 迁移旨在实现NSManagedObjectModel中的更改,而不是数据本身。
What you really need is a Pesristent Store Coordinator managing two Persistent Stores. That's kinda tricky, but not too difficult, really.
你真正需要的是一个管理两个持久商店的Pesristent商店协调员。这有点棘手,但并不太难,真的。
There's a similar question, that can explain you,what you really need to do. Can multiple (two) persistent stores be used with one object model, while maintaining relations from one to the other?
有一个类似的问题,可以解释你,你真正需要做的。多个(两个)持久性存储可以与一个对象模型一起使用,同时保持从一个到另一个的关系吗?
Here's a good arcticle by Marcus Zarra
这是Marcus Zarra的一个很好的arcticle
#1
6
After several attempts, I've figured out how to make this work. The secret is to first create the incremental store data without any data for the read-only entities. Without leaving read-only data out of the incremental stores, the entities instances for these would get duplicated after the data migration and merge. Hence, the incremental stores should be created without these read-only entities. The default store will be the only store that has them.
经过几次尝试,我已经想出了如何使这项工作。秘诀是首先创建增量存储数据,而不包含只读实体的任何数据。如果不从增量存储中删除只读数据,则在数据迁移和合并之后,这些实体的实体实例将会重复。因此,应该在没有这些只读实体的情况下创建增量存储。默认商店将是唯一拥有它们的商店。
For example, I had entities "Country" and "State" in my data model. I needed to have only one instance of Country and State in my object graph. I kept these entities out of incremental stores and created them only in the default store. I used Fetched Properties to loosely link my main object graph to these entities. I created the default store with all the entity instances in my model. The incremental stores either didn't have the read-only entities (i.e., Country and State in my case) to start with or deleted them after data creation is completed.
例如,我的数据模型中有实体“Country”和“State”。我需要在对象图中只有一个Country和State实例。我将这些实体保留在增量存储之外,并仅在默认存储中创建它们。我使用Fetched Properties将我的主对象图松散地链接到这些实体。我在模型中创建了包含所有实体实例的默认存储。增量存储在数据创建完成后没有只读实体(即我的国家和国家)开始或删除它们。
Next step is to add the incremental store to it's own persistentStoreCoordinator (not the same as the coordinator for the default store that we want to migrate all contents to) during application startup.
下一步是在应用程序启动期间将增量存储添加到它自己的persistentStoreCoordinator(与我们要将所有内容迁移到的默认存储的协调器不同)。
The final step is to call migratePersistentStore method on the incremental store to merge its data to the main (i.e., default) store. Presto!
最后一步是在增量存储上调用migratePersistentStore方法,将其数据合并到主(即默认)存储中。普雷斯托!
The following code fragment illustrates the last two steps I mentioned above. I did these steps to make my setup to merge incremental data into a main data store to work.
以下代码片段说明了我上面提到的最后两个步骤。我做了这些步骤,使我的设置将增量数据合并到主数据存储中。
{
NSError *error = nil;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption,
[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];
if (![__persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:defaultStoreURL options:options error:&error])
{
NSLog(@"Failed with error: %@", [error localizedDescription]);
abort();
}
// Check for the existence of incrementalStore
// Add incrementalStore
if (incrementalStoreExists) {
NSPersistentStore *incrementalStore = [_incrementalPersistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:incrementalStoreURL options:options error:&error];
if (!incrementalStore)
{
NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
abort();
}
if (![_incrementalPersistentStoreCoordinator migratePersistentStore:incrementalStore
toURL:_defaultStoreURL
options:options
withType:NSSQLiteStoreType
error:&error])
{
NSLog(@"%@", [error userInfo]);
abort();
}
// Destroy the store and store coordinator for the incremental store
[_incrementalPersistentStoreCoordinator removePersistentStore:incrementalStore error:&error];
incrementalPersistentStoreCoordinator = nil;
// Should probably delete the URL from file system as well
//
}
}
#2
1
The reason your migration isn't working is because the managed object model is identical.
迁移不起作用的原因是托管对象模型相同。
Technically, you're talking about "data migration" not "schema migration". CoreData's migration API is designed for schema migration, that is handling changes to the managed object model.
从技术上讲,您所说的是“数据迁移”而不是“模式迁移”。 CoreData的迁移API专为模式迁移而设计,即处理对托管对象模型的更改。
As far as transferring data from one store to another you're kind of on your own. CoreData can help you be efficient by using batching and fetch limits on your fetch requests, but you need to implement the logic yourself.
至于将数据从一个商店转移到另一个商店,你就是自己的一种。 CoreData可以通过对获取请求使用批处理和获取限制来帮助您提高效率,但您需要自己实现逻辑。
It sounds like you have two persistent stores, a big one and a small one. It would be most efficient to load the small one and analyze it, discovering the set of primary keys or unique identifiers you need to query for in the larger store.
听起来你有两个持久性商店,一个大商店和一个小商店。加载小的并分析它,发现在较大的商店中需要查询的主键集或唯一标识符是最有效的。
You could then de-dupe easily by simply querying the larger store for those identifiers.
然后,您可以通过简单地查询较大的商店来获取这些标识符来轻松删除。
The documentation for NSFetchRequest has the API for scoping your queries:
NSFetchRequest的文档具有用于确定查询范围的API:
#3
1
You don't need any migration - migration is designed to bring changes in NSManagedObjectModel,not in data itself.
您不需要任何迁移 - 迁移旨在实现NSManagedObjectModel中的更改,而不是数据本身。
What you really need is a Pesristent Store Coordinator managing two Persistent Stores. That's kinda tricky, but not too difficult, really.
你真正需要的是一个管理两个持久商店的Pesristent商店协调员。这有点棘手,但并不太难,真的。
There's a similar question, that can explain you,what you really need to do. Can multiple (two) persistent stores be used with one object model, while maintaining relations from one to the other?
有一个类似的问题,可以解释你,你真正需要做的。多个(两个)持久性存储可以与一个对象模型一起使用,同时保持从一个到另一个的关系吗?
Here's a good arcticle by Marcus Zarra
这是Marcus Zarra的一个很好的arcticle