I've been importing data into my Cloud SQL instance using the gcloud import tool. I've broken up the data into smaller chunks to avoid the 24 hour limitation on imports. (Each import has about 16 million rows grouped into about 16 thousand import statements.) I completed two of the imports (each taking about 2 hours) and was importing the third:
我一直在使用gcloud导入工具将数据导入我的Cloud SQL实例。我将数据分解成较小的块以避免24小时的进口限制。 (每个导入有大约1600万行,分为大约16000个导入语句。)我完成了两个导入(每个大约需要2个小时)并导入第三个:
gcloud sql instances import --database mydb mydb-1 gs://data-1/olddata/olddata-2010.sql.gz
After running for about 90 minutes, it stopped and reported the error
运行约90分钟后,它停止并报告错误
ERROR: (gcloud.sql.instances.import) Daily Limit Exceeded
I've found references to a 24 hour limit per import, but no other "daily limit" documented related to imports that I can see.
我发现每次进口都提到24小时限制,但没有其他“每日限制”记录与我可以看到的进口有关。
If I look at the API portion of the console, I do see that the Google Cloud SQL API has a daily limit of 10,000 requests/day, and that I have exceeded this. I can't reconcile this with my use of any of the Cloud SQL tools - either I would have had to do far more than 10000 requests, or far fewer. Looking at the Cloud SQL API reference documentation for v1beta4 the Instances.import documentation doesn't say it should take more than a single request to run (in fact, it doesn't say anything at all about quota). There don't appear to be any daily limits associated with retrieving from Cloud Storage.
如果我查看控制台的API部分,我确实看到Google Cloud SQL API的每日限制为每天10,000个请求,而且我已超出此限制。我无法通过使用任何Cloud SQL工具来协调这一点 - 要么我要做的远远超过10000个请求,要么远远少于10000个。查看v1beta4的Cloud SQL API参考文档,Instances.import文档并没有说它应该运行多个请求(事实上,它没有说任何关于配额的内容)。从云存储中检索似乎没有任何每日限制。
- What is the daily limit?
- Where is the daily limit documented?
- Is there a way to get this limit increased?
- How many API calls to the Cloud SQL API are actually done for an import?
- Is there a way to do the import without running into this limit? (I have about 10 more data sets to import and would rather not have to space them out.)
每日限额是多少?
记录的每日限额在哪里?
有没有办法让这个限制增加?
实际为导入执行了多少次对SQL SQL API的API调用?
有没有办法在没有达到此限制的情况下进行导入? (我还有大约10个要导入的数据集,而不必将它们分开。)
1 个解决方案
#1
Although you can ask to have your quota increased - see this page (center section) -- GAE quotas, it may not be needed. What sometimes mitigates the situation, is for you to examine exactly how the grouped data is being processed. Are you using any sort of trace tool or looking at the logs?
虽然您可以要求增加配额 - 请参阅此页面(中心部分) - GAE配额,但可能不需要。有时可以缓解这种情况的是,您可以准确地检查分组数据的处理方式。您使用任何类型的跟踪工具或查看日志?
I have found that manually tuning the batch size based on these results did help in this case. Also, is this a one-time import? or it is expected to be repeated?
我发现在这种情况下,根据这些结果手动调整批量大小确实有帮助。另外,这是一次性导入吗?或者预计会重复?
