Importing/Exporting NDB Data on GAE: a quick scripting solution

Importing and Exporting database data is a common and sometimes quite trivial in most frameworks. SQL databases and cloud platform usually have good tools that would allow the developer / dev ops / DBA to replicate the data between environments without having to write one line of code. Unfortunately, this is not the case for Python on Google App Engine.

Reading around the docs and some stack overflow threads, it looks like there used to be an option to export and import data in an older version of GAE with the "master/slave datastore."

However with the high replication data store this method no longer works, and one has to figure out the easiest way to import/export data on his own.

Importing/Exporting data in any production environment is a critical task: even if you don't need to export the data for reporting and analysis, you would still want to have similar data in your dev, test and staging environments.

One way of doing this quite easily is writing some code to fetch all records, iterate over the collection, and script the result as a python script that would perform the insert.


This bit for example will create a python script to insert all "campaign" entity instances:

class export_db(webapp2.RequestHandler):
    def get(self):
        text = ''
        all_campaigns = models.campaign.query().fetch(200)
        for i in all_campaigns:
            text += 'c = models.' + str(i).replace('Key(', 'ndb.Key(') + '\n'
            text += 'c.put()\n'

The result text can then be pasted into a function that will perform the inserts one-by-one:

class import_db(webapp2.RequestHandler):
    def get(self):
        c = models.campaign(key=ndb.Key('campaign', 'April Campaign'), created_on=datetime.datetime(2014, 4, 30, 8, 20, 5, 674460), end_date=datetime.date(2014, 10, 10), modified_on=datetime.datetime(2014, 4, 30, 8, 20, 5, 674470), name=u'App Center April Campaign', start_date=datetime.date(2014, 4, 1), target_app=u'app-center')
        c.put()

Watch out for unicode chars though. Although they will be scripted automatically using the "u" prefix, you would still have to decode them for the insert to work.