How To Export Firestore Data To JSON File: Complete Guide

Firestore is known for being a solid cloud-based database that’s widely used for storing and synchronizing data for applications. Its popularity is no coincidence – it has consistently proven to be a flexible and reliable choice for data storage. For this reason, or whether it’s for analysis, backup, or usage in other systems, developers and businesses often need to export data from Firestore to JSON.

However, there is a challenge – Firestore does not offer an immediate option within the Google Console to export data as a JSON file. Fortunately, there are alternative methods to perform this task.

In this guide, we will outline 3 straightforward methods for exporting Firestore data to a JSON file. By the end of this 8-minute read, you will gain a clear understanding and learn the necessary steps to export Firestore data to a JSON file.

What Is Firestore?

Image Source

Firestore, or Cloud Firestore as it’s officially known, is Google’s NoSQL database that is specifically designed to store and synchronize data. This synchronization is maintained for both client and server-side development. This proves to be highly beneficial, especially for mobile and web applications.

Firestore is extensively adopted across various domains, including mobile applications, web services, and server-side systems. Common use cases comprise real-time chat applications, content management systems, user profiles, and online marketplaces.

Some of the key features of Firestore include:

• Security: Data security is a priority in Firestore. It allows for setting up rules to control access to the data and ensures that sensitive information is properly protected.
  
• Scalability: It can handle increasing amounts of data and traffic. This is crucial for growing applications as it takes the burden of scaling the database off the developer.
  
• Structured data: Firestore uses documents for storing data. These documents are grouped into collections. Each document is a set of key-value pairs and can also contain subcollections.
  
• Offline capabilities: It can store data locally on the device when there is no internet connection. Once the connection is restored, it automatically syncs the local data with the cloud.
  
• Real-time synchronization: One of the standout features of Firestore is its ability to sync data across different devices in real time. If data changes on one device, the update is immediately available on all other devices.

Understanding Firestore Data Model

Image Source

There are 3 fundamental elements in the Firestore data model.

1. Documents: The fundamental units for storing data in Firestore are documents. Each document is a set of key-value pairs.
   
2. Collections: Documents are organized into collections which serve as a way to categorize data.
   
3. Subcollections: Within a document, you can have additional collections known as subcollections. This allows for a hierarchical data structure. For example, in a “Users” collection, a user document might have a subcollection called "orders" representing that user’s orders.

A very important aspect of Firestore is that it is schemaless, meaning that there's no predefined structure that documents must adhere to. Documents within the same collection can have different fields and data types.

What Is The JSON Format?

JSON, which stands for JavaScript Object Notation, provides a simple way to represent data structures and is widely used in web applications for data exchange. It was originally conceived to streamline the communication between JavaScript clients and servers. However, developers quickly realized JSON's flexibility and started to use it for other purposes. 

JSON’s immense popularity can be attributed to its lightweight nature and human-readable format. It is also flexible because almost every programming language supports it. Due to its widespread compatibility, JSON has become the de facto standard for exchanging data. 

In this section, we will explore the basic JSON structure and discuss important details that will enhance your understanding of this widely used data format.

• Key-value pairs: JSON organizes data into key-value pairs. The key is the name of the item and the value is the data related to that name. So if you have a person named John who is 30 years old, in JSON, it would look like: {"name": "John", "age": 30}.
  
• Hierarchy: Like Firestore, JSON can also have a hierarchy. Values can be more than just numbers or words; they can also be groups of values (which we call arrays) or even other pairs of keys and values (which we refer to as objects).
  
• Arrays: Arrays are lists of values that are wrapped in square brackets in JSON. For instance, a list of numbers would look like this: [1, 2, 3].
  
• Syntax essentials: JSON follows specific rules for its written format, including:
  
  • Keys must be strings (text) and they have to be in double quotes.
  • Values can be different data types like numbers, strings, booleans, arrays, or other JSON objects.
  • Key-value pairs are separated by commas.
  • The whole JSON data is usually wrapped in curly brackets {}.

3 Proven Methods To Export Data From Firestore To JSON

Exporting data from Firestore to a JSON file is required for several reasons – creating backups, facilitating data migration, or enabling data sharing and local analysis. JSON is the ideal choice for these purposes because it is widely compatible and easy to read. 

However, Firestore poses a problem in this regard. As it turns out, Firestore does not natively support exporting data in JSON format. Instead, the default Firestore export functionality is confined to a binary format stored in Google Cloud. You can read more about the Firestore export/import functionalities here.

While binary format serves certain use cases, it lacks the flexibility and accessibility that JSON offers. But there are options available to get the job done. 

Let's look at three different methods to export Firestore data to JSON.

Prerequisites

Follow the steps given below before trying to export Firestore data. The first 2 steps are only needed if you are setting up a new database. If your database is already up and running, jump directly to step 3.

Step 1: Setting Up A Firebase Project

First, you need a Firebase project. This is the foundation where all your data is stored.

1. Go to the Firebase console.
   
2. Click on ‘Add project’.
   
3. Fill in the project name and follow the on-screen instructions.

With this, you now have a Firebase project.

Step 2: Enabling Firestore In The Firebase Console

Now let’s enable Firestore.

1. In your Firebase project, select ‘Cloud Firestore’ in the left-hand menu.
   
2. Click on ‘Create database’.
   
3. Choose either ‘Start in production mode’ or ‘Start in test mode’, depending on your needs.
   
4. Select a Cloud Firestore location (this is important for latency and availability).

Step 3: Setting Up A Service Account & Permissions

To access and export data from Firestore, set up a service account and generate a key for it. Additionally, this account should have the required permissions.

1. In the Google Cloud console, head to the Create service account page.
2. Select your Google Cloud project.
   
3. Input a name for the service account.
   
4. Optionally, enter a description.
   
5. Click ‘Create and continue’ for access controls or ‘Done’ if you prefer setting them later.
   
6. Optionally, grant one or more IAM roles to the service account.
   
7. Click ‘Continue’.
   
8. Optionally, add members who can impersonate or manage the service account.
   
9. Click ‘Done’ to finalize creating the service account.

Now let’s generate a new private key for your service account.

1. On the Service Accounts page, select your project.
   
2. To generate a key for a specific service account, select its email address.
   
3. Navigate to the Keys tab.
   
4. Select ‘Create new key’ from the ‘Add key’ drop-down menu.
   
5. Choose JSON as the Key type and click ‘Create’.

This action downloads a service account key file. IMPORTANT: Keep it safe as you can’t download it again.

Method 1: Exporting Real-time Data From Firestore Via Estuary Flow

Estuary Flow is our advanced DataOps platform for handling real-time data operations through managed CDC (Change Data Capture) and ETL (Extract, Transform, Load) pipelines. Estuary Flow offers powerful features, such as performing complex real-time transformations and aggregations, allowing for rapid data transfer with end-to-end latency in the millisecond range.

Flow offers several benefits, including:

• Scalability: It can manage large-scale active workloads, even at rates of 7GB/s, catering to databases of any size.
  
• Extensibility: Flow supports integration with various connectors through an open protocol and includes built-in testing.
  
• Automation: There is no need for scheduling or orchestration tools. Flow offers automated schema management and data deduplication.
  
• User-friendly interface: Flow stands out for its user-friendly interface, providing an intuitive experience that simplifies data processing and transfer tasks, even for users with limited technical expertise.
  
• Cost effective: By using CDC, Flow minimizes the data read from the data source and the data written to the destination, which makes it an economical solution.
  
• Real-Time data transfer: Flow ensures that your data is transferred immediately as changes occur in Firestore. This real-time data processing ensures that the data remains current.

Now, let's turn our attention to the task at hand: importing data from Firestore into Google BigQuery. Once the data is in BigQuery, you can export the data in JSON format. With Estuary Flow’s powerful, automated CDC and real-time capabilities, this process is both streamlined and cost-effective.

Step 1: Log In To Estuary Flow

Navigate to the Estuary Flow dashboard and sign in. If you don't have an account, sign up for one for free.

Step 2: Create A Flow Capture From Firestore

1. Click on the ‘Captures’ tab and select ‘New capture’.
   
2. Look for ‘Google Firestore’ and select it as the connector.
   
3. Type a unique name for your capture.
   
4. Under Endpoint Config, Input your Google Cloud Service Account JSON credentials into the ‘Credentials’ field.
   
5. (Optional) You can specify the database name in the 'Database' field or leave it blank for autodetection.
   
6. Click ‘Next’. Estuary Flow will connect to Firestore and identify available data resources. You’ll be prompted to choose the collections you want to capture.
   
7. Review the configurations and when you're happy with them, click on ‘Save and publish’. A notification will pop up once the capture is published. 

Important Note: Estuary Flow natively stores all data in JSON format, ensuring that your captured collections are neatly organized in your cloud storage. To seamlessly connect your cloud storage to Estuary Flow, you can refer to this helpful guide.

Consolidating the data from BigQuery into a single, larger file is another approach. This combines all the data into a single, larger file instead of dealing with multiple smaller files. If you prefer this method, let's proceed to the next step.

Step 3: Create A Flow Materialization To BigQuery

1. There are a few prerequisites you need to satisfy before you can use Flow to connect to BigQuery. Before moving forward, please make sure these requirements are met. 
   
2. Next, go to the ‘Materializations’ tab and click on ‘New Materialization’.
   
3. Now select ‘Google BigQuery’ as the connector.
   
4. Choose a unique name for your materialization, similar to what you did for the capture.
   
5. Fill in the required configuration settings under Endpoint Config.
   
6. The Source Collections section now shows each collection you captured earlier. Here you can make changes to the collections before materializing them.
   
7. Click ‘Next’.
   
8. Once satisfied with the configuration, click on ‘Save and publish’.

After you complete these first three steps, Flow will start moving your Firestore data to BigQuery in real time. You can analyze the data in BigQuery or download it as a JSON file by following the next step. 

Step 4: Download Data From BigQuery In JSON format

Once you have transferred your data to Google BigQuery, you can download it for local analysis or backup. Here are the steps to download data from BigQuery into JSON format:

Export Table To Google Cloud storage

BigQuery only allows exporting data to Google Cloud storage. We will first move our JSON file there.

1. Go to the BigQuery Console.
   
2. Find the table you want to export and click on it.
   
3. Click on “Export”, then choose “Export to Cloud Storage”.
   
4. Choose JSON as the export format.
   
5. Provide the Google Cloud Storage location.
   
6. Click on “Export”.

Download The Exported Data To Your Local System

Now you can download the JSON data exported to Google Cloud Storage to your local system.

1. Open the Google Cloud Console.
   
2. Find the bucket where you exported the BigQuery table.
   
3. Navigate to the file you want to download.
   
4. Click on the filename and then click on the “Download” button.

Method 2: Manually Exporting Firestore Data Using Firebase Admin SDK & Python

Let’s now look at the process of exporting data from your Firestore database to a JSON file using Python. While this method is straightforward and effective for small-scale data exports, it’s important to be aware of its limitations.

• First, this approach is not scalable for large datasets as retrieving and processing a massive amount of data can be resource-intensive and time-consuming. 
  
• Second, this approach might not be ideal for continuous or automated data syncing due to its manual nature. 

Keeping these factors in mind, let’s explore the step-by-step process of exporting data from your Firestore database to JSON format using Python.

Step 1: Prerequisites

Before you begin, make sure you have the following:

• Python 3 installed on your system.
  
• A Firebase project with a Firestore database set up.
  
• Service account key file generated from your Firebase project.

Step 2: Install The Firebase Admin SDK

You’ll need the Firebase Admin SDK to interact with Firestore using Python. Run the following command to install it:

pip install firebase-admin

Step 3: Authenticate With Firebase Admin SDK

Now you need to authenticate with Firebase using your service account key file. Replace the key_file variable with the path to your service account key file.

import firebase_admin

from firebase_admin import credentials

key_file = "/path/to/your/service-account-file.json"

cred = credentials.Certificate(key_file)

firebase_admin.initialize_app(cred)

Step 4: Initialize Firestore Client

Initialize the Firestore client to start interacting with the Firestore database.

from firebase_admin import firestore

firestore_client = firestore.client()

Step 5: Retrieve Data From Firestore

Specify the collection you want to export data from and retrieve its documents. For this example, let’s say the collection’s name is ‘laptops’.

collection_name = 'laptops'

documents = firestore_client.collection(collection_name).stream()

Step 6: Prepare Data For JSON export

Loop through the documents and store them in a dictionary.

data = {}

for doc in documents:

    data[doc.id] = doc.to_dict()

Step 7: Write Data To A JSON File

Finally, write the data to a JSON file. You can specify any file name you want.

import json

json_file = 'firestore_data.json'

with open(json_file, 'w') as file:

    json.dump(data, file, indent=4)

print(f"Data has been exported to {json_file}")

Method 3: Manually Exporting Firestore Data Using A Third-Party Tool

Let’s explore the last approach for exporting Firestore data to a JSON file using a third-party tool, Firefoo. It is a desktop tool that is available on Mac, Windows, and Linux. Not only does it allow you to manage multiple Firebase projects, but it also lets you efficiently browse, query, and edit your Firestore data. 

Before we jump into the steps of exporting Firestore data using Firefoo, let’s discuss some of its limitations. 

• One of the significant drawbacks of using Firefoo is that you need to export data manually. It doesn’t offer an automated way to handle regular backups or exports. 
  
• Secondly, you need to purchase a license to continue using it beyond the trial period.

Now that we’re clear on what to expect, let’s get down to business. Here’s a step-by-step guide to exporting Firestore data using Firefoo:

Step 1: Download & Install Firefoo

1. Go to the Firefoo website and download the tool.
   
2. Follow the installation instructions to set it up on your system.

Step 2: Connect Firefoo To Firebase

1. Open Firefoo. If you’ve used Firebase CLI tools before, Firefoo might automatically connect to your project.
   
2. If not, you'll see the "Add Firebase Project" popup.
   
3. You have 3 options: Sign in with Google, Import Service Account Key JSON, or Add Local Emulator.

Step 3: Select Your Firebase Project

1. Once you’re signed in, close the browser tab and head back to Firefoo.
   
2. You should now see a list of Firebase projects in the sidebar. Select the one you want to export data from.

Step 4: Export Firestore Data To JSON

Image Source

1. Right-click on an existing Firestore collection you want to export.
   
2. Select "Export Collection".
   
3. A dialog box will pop up. Make sure "JSON Format" is selected.
   
4. Click the Export icon to begin the export operation.
   
5. Choose where you want to save the file and you're good to go.

Image Source

Step 5: Check The Progress

1. A progress popup will appear. You can close it; the export will continue in the background.
   
2. To see the progress again, go to File > Tasks.

Conclusion

Exporting data from Firestore to JSON is a vital skill for cloud database administrators, developers, and data engineers seeking efficiency, scalability, and data portability. This process empowers you to break free from vendor lock-in and ensures that your data remains accessible and usable even if you decide to switch to a different database system in the future. 

Exporting Firestore data to JSON gives you the freedom to integrate your data with various platforms, perform in-depth analysis, and build custom data pipelines tailored to your specific needs.

Flow stands out as the best option and offers a user-friendly and reliable platform for exporting Firestore data to JSON files. Its robustness, usability, and scalability make it ideal for professionals who are focused on simplifying data management and transfer. The capability to handle real-time data and Firestore CDC are the key advantages that Estuary Flow brings to the table.

So, if you're looking for a streamlined solution to facilitate the effortless export of Firestore data, Flow is here to cater to your needs. Sign up now and get started for free or reach out to our team for more information on how our platform can be tailored to your specific requirements.