Data import

[Vortec4800]

I know on the hosted Parse dashboard (non-beta) there's an option to import data via JSON or CSV which is great. I haven't been able to find this option on the hosted beta dashboard, or on the parse-dashboard project.

Is this functionality available in either place? Is it a planned addition to the dashboard project, or out of scope? Seeding in data from a CSV file is a really useful feature and I was sad to not find it.

[neophob]

should be obselte now as you can import the data with mongoimport. or are there other advantages when importing via webui?

[Vortec4800]

That's a good and valid point. I've been treating the mongo database like a black box, so I don't mess with the way the Parse server is storing data. I suppose it wouldn't be crazy to import directly to the database.

I do still think having the capability in the dashboard would be useful from a UX standpoint. It's much easier in general and much harder to screw up. I like to hand the keys to a project over to a client, and lately that's been including a Parse dashboard login with instructions on how to do basic data management through there. It would be great if they could import data to their databases via CSV on the dashboard without needing us to get involved on the database itself. The more my clients and project stakeholders can manage on their own, the better for me.

[davimacedo]

I think it should be added to the dashboard by the community. My team can work on it in the future. Meanwhile, you can use cloud functions to import data. I wrote an example how you can do it:
https://gist.github.com/davimacedo/c7738203a2a616b48a22fdc7c00b37d2

[JoaqStar]

I also used the import feature to import csv to my parse database. I looked up mongoimport and do not understand how it works. Can you please let me know what is the easiest way to import csv data to my parse table?

[sighjones]

I think the removal of the import function on the dashboard is extremely short sighted. As Vortex rightly points out, giving the client the ability to import simple csv data is EXTREMELY useful. Can you give any idea of a timeline for adding it back in?

[drew-gross]

mongoimport can also work with CSV. Check the docs here: https://docs.mongodb.com/manual/reference/program/mongoimport/

[dailyneedsapp]

Yes definitely the import data via CSV/Json should be added back to Parse Dashboard.

[mitchellporter]

+1 for adding it back. Was very easy to use, even for someone that doesn't code.

[ndmeiri]

+1

[kikrlabs]

Hi, I don't see the import data via CSV/Json, was this added back? I accidentally deleted a class with 98 rows. Please add this button back as my data is in json format and need to import my data soon.

[RafaRuiz]

Yes they removed the button even on the non-beta dashboard :(

[KapoorKarn]

+1 This function was crucial to helping build apps fast.

[KapoorKarn]

I'm down to put a bounty on this feature. Anyone else want to join to see if we can get it large enough for someone to tackle?

[miracle7]

+1, can't believe I can no longer do this with self hosted dashboard.

[stretchnz]

+1 I would very much like this feature, more or less instant CSV import was great for our use case, now uploading data using node (in the legacy .com system) which takes considerable time to complete due limits on connection rates etc.

[KapoorKarn]

All, take a look at mongochef (http://3t.io/mongochef/), i've been using it since the removed the bulk import function from the dashboard. Mongochef has allowed me to easily add/remove/modify classes.
I've been able to import semi large csv's to my dbs (roughly 10k records). For large csv files i've been doing a bulk import. There is a commercial and non-commercial option available.

I know some other have mentioned some ways to import as well, but i'm not very skilled in scripts, shells, or yadda yadda. Anyway, just wanted to post in case it helped someone.

[Knana]

NodeChef has rolled out a feature that allows you to easily import CSV and JSON data into your Parse Server database hosted on NodeChef with just a click of a button. Documentation available here.

[zh-wowtv]

Without this, how are we suppose to import my previous data? Directly operating on mongodb is never a good idea.

[Treverr]

Any update on this? Or know of any options, I've tried the cloud function, it doesn't work anymore.

[AmrKafina]

@Treverr I recently set up a simple tool that automates the process. JSON or CSV formatted data is supported.

[nyxee]

@AmrKafina, how can we use the tool for local databases please???

[ebelinski]

@AmrKafina your link is dead.

[montymxb]

Hey everyone, we're not focused on this atm (importing straight to mongodb is usually the way to go) but I too would like to see this back in place :) , it was a nice feature. Considering this would be a convenience, but not a critical component, we'll probably get to this after handling other issues with the dashboard first.

We're totally open to anyone who wants to get started on this in the meantime. It would be prudent to note that if we're going to add an import function we're going to want to also add a compatible export function.

[Macstyg]

@montymxb I have exported .json files from Parse Dashboard, from Back4App service. In their dashboard the import/export functionality is still there. When I try to do mongoimport from .json file to my local DB everything is importing fine. And I can see that data is my database through the mongo shell. But in ParseDashboard I don't see the new collections. Can you know any possible solutions for this problem?

[montymxb]

@Macstyg Hmmm, that doesn't sound quite right. I haven't worked with migrating to (or from) back4app personally so I can't attest to the process there. I would reach out to their support and see if there's something you're missing in order to do a complete import. Although it should just work. A recent migration I oversaw away from a 3rd party provider went relatively smooth via mongoimport.

[wimbledon]

@Macstyg back4app server settings has mongo db connection string with the credentials. You can use that to connect to your back4app's mongo db and use mongoimport to import your json data that way.

[meirka]

It is very handy tool, is there a progress or is it on roadmap?

[tanzeelrana]

this feature is a must !! any luck anyone ?

[Sbef98]

We're still waiting! If i'm lucky i may eventually provide it myself, but this is a seriously short sighted move.

[mtrezza]

I think we could provide at least a very basic version for data import.

It should accept the same format as the existing data export feature, i.e. it should be possible to export a row, delete it from the DB then re-import it without having to modify the data.

The basic version would not cover:

• Automatically or manually resolve any conflicts, e.g. importing an object with an objectId that already exists
• Batched import, i.e. importing large data sets my have a neg. impact on the database performance

Open questions:

• How would that data import work? It should not invoke any Cloud Code triggers, so we can't just save the object. Parse Server will likely need a new interface that allows to import directly to the DB, bypassing a few mechanisms.

[Sbef98]

• How would that data import work? It should not invoke any Cloud Code triggers, so we can't just save the object. Parse Server will likely need a new interface that allows to import directly to the DB, bypassing a few mechanisms.

Why should it not activate the triggers? What would it be the perks of implementing such import at that point? In my perspective, the advantage of importing data directly from parse dashboard, apart from the ease of use, would be exactly that my triggers would assure that the data imported are consistent with the policies i'm implementing.

Especially in the scenario in which it's the customer/user the one who is importing datas, allowing him to do it through the dashboard but still forcing him to comply to the implemented policies through the triggers would be a beautiful plus to me!

In such case, even if something is going to change during the migration/import process, such changes are done consistently to the same rules i'd apply to fresh data! Setting new ACL policies, setting new data consistency checks etcetera. This would somehow improve the import also for these who want to move, for example, from an Acess dataset to a Parse istance (and similar cases).

[mtrezza]

That is an interesting take on data import. I don't think that such a behavior would be widely expected though. Data import / export are usually understood as being idempotent operations. Maybe this could be an optional behavior, but it can get pretty complicated, as import order suddenly becomes important, etc.

Here are 2 uses cases:

• a) A user wants to bulk import / update products in a store: they use the import data as a replacement for manually adding rows in the data browser; each imported product should be validated and invoke triggers, just like it had been entered manually; the import "mimics" a manual data entry, so the order of data imported (also across different classes) becomes important
• b) A developer wants to restore data in their development environment: they just want to reset all rows in a collection to a previous state, so they delete all the modified data and re-import all the initial data; they don't want any trigger to be invoked, because the initial data is already in its final state

[dblythy]

How should this handle objectId, createdAt, and updatedAt? Currently in Parse Server these fields are read-only.

[mtrezza]

The answer lies in which feature we want, see #307 (comment).

I would choose whatever is easiest and requires the least change.

It seems there are 2 restrictions that are Parse Server-side:

• objectId, createdAt, updatedAt are readonly and cannot be imported
• triggers being invoked

If overcoming these restriction require a Parse Server PR, we probably should focus on use case (a).

Another supporting argument for (a) is that an import without the aforementioned restrictions is already possible via a raw database import, using MongoDB or Postgres tools.

[dblythy]

Option a requires the least effort, but I would assume if objectId is defined in the payload, it would be expected that it is respected. Otherwise it makes it difficult to import pointers etc.

[mtrezza]

We can approach this incrementally and start with a basic version with limitations.

Basic version could do only import of data according to use case (a).

Future feature improvements:

• Option to modify pointers on-the-fly during import
• Option whether to invoke class triggers during import
• Option to import data as-is (incl. read-only fields like createdAt, updatedAt, objectId), which is use case (b)

These improvements will likely require changes in Parse Server as well. Therefore I'd suggest to take an incremental approach to down-size the scope and complexity of each PR.

[dblythy]

Sounds like a plan to me!

[JuYiYang]

This feature really shouldn't be cut

[mtrezza]

Bumped bounty given the broad interest in this feature.