Heracles

What is Heracles?

This repository provides a library for encoding a Hydra 3D scene graph into a Neo4j database. It expects a scene graph to initially be loaded from file via spark_dsg and provides utilities for translating back and forth between spark_dsg and the Neo4j database.

We provide three examples that should be pretty easy to run. They require that Docker is installed, but they should have no other dependencies.

Minimal Example

To run the first example, run

./heracles_demo.sh

This script will load an example scene graph into the graph database and put you in a python terminal where you can run example queries. For example, try running the following query to print the location of all trashcans:

db.query("MATCH (n: Object {class: 'trash'}) RETURN n.center as center""")

Interactive ROS Example

The second example will render the current state of the scene graph in the database in Rviz. You do not need ROS installed to run this demo. However, you will need to specify an OpenAI api key HERACLES_OPENAI_API_KEY if you want to use the LLM interaction functionality.

export HERACLES_OPENAI_API_KEY=<YOUR OPENAI API KEY>
xhost +local:docker && ./chatdsg_ros_demo.sh

Try asking the LLM how many objects are in the scene graph (press ctrl-b to send your message to the LLM). Then try telling it to update some of the objects (e.g., turn signs into lights).

Interactive Viser Example

Finally, we provide an example of visualizing the scene graph using Viser, so that no ROS is necessary. Again, you will need an OpenAI API key if you want to use the LLM functionality. Make sure to go to http://127.0.0.1:8081 in your browser after running the script!

export HERACLES_OPENAI_API_KEY=<YOUR OPENAI API KEY>
./chatdsg_viser_demo.sh # Then navigate to http://127.0.0.1:8081 in your browser

You may have to pan the camera around if the scene graph is rendered out of the initial camera frame.

Development Setup

If you want functionality beyond these basic demos, you will want to install heracles locally. This requires 1) spinning up a database, and 2) installing `heracles.

Pull the Neo4j database image:

docker pull neo4j:5.25.1

Run the database:

docker run -d \
   --restart always \
   --publish=7474:7474 --publish=7687:7687 \
   --env=NEO4J_AUTH=neo4j/neo4j_pw \
   neo4j:5.25.1

Clone and install this repo:

git@github.com:GoldenZephyr/heracles.git
cd heracles
pip install .

If you run into installation problems, check .github/workflows/ci-action.yml for an example, as that successfully installs and runs the library for CI.

If you don't have a scene graph to test with, you can use a large example scene graph here. This scene graph has 2D places and objects, and 3D places (that don't make much sense), but no regions or rooms. This repo contains an example scene graph as well.

Demo - Loading a Scene Graph

We provide a demo program that loads a scene graph from file, encodes it into a database, and performs a simple query for validation. The demo program is implemented with interactive mode in mind to make it easy to try other kinds of queries.

Environment Variables

We expect certain environment variables to be set, listed below. HERACLES\_NEO4J\_URI can alternatively be provided via the command line. An example of HERACLES\_NEO4J\_URI is neo4j://127.0.0.1:7688.

Environment Variable Name	Description
HERACLES_NEO4J_USERNAME	Username of local Neo4j graph database
HERACLES_NEO4J_PASSWORD	Password of local Neo4j graph database
HERACLES_NEO4J_URI	Address for database (neo4j://IP:PORT)

Run the program

To run the vanilla demo (load a scene graph and print a summary):

cd heracles/examples/
./load_scene_graph.py

The demo is useful to run in interactive mode. We leave the DB connection open, so it is possible to easily try other queries. To run the program in interactive mode:

cd heracles/examples/
python -i load_scene_graph.py

The demo should run as normal but end in an active interactive mode. You can immediately try another query. For example:

db.query("MATCH (n: Object) RETURN DISTINCT n.class as class, COUNT(*) as count")

Heracles ROS

If you would like to use heracles with ROS, heracles_ros provides an interface for publishing the scene graph from the graph database as a ROS message. If you would like to use this functionality, you should clone this repo in your ROS workspace. Instead of manually pip installing as described above, you can let colcon take care of the installation (as long as your virtual environment is created with --system-site-packages). If you also have Hydra installed, you can visualize the scene graph that's in the graph database with the following command:

ros2 launch heracles_ros demo.launch.yaml launch_heracles_publisher:=true launch_hydra_visualizer:=true launch_rviz:=true

Useful Queries

Print distinct object classes in scene graph

db.query("MATCH (n: Object) RETURN DISTINCT n.class as class""")

Print distinct object classes and counts

db.query("MATCH (n: Object) RETURN DISTINCT n.class as class, COUNT(*) as count""")

Print center point of objects of specific class

db.query("MATCH (n: Object {class: 'tree'}) RETURN n.center as center""")

Filter for objects of a certain type within a bounding box

db.query("""WITH point({x: -100, y: 16, z: -100}) AS lowerLeft,
                 point({x: -90, y: 22, z:100}) AS upperRight
            MATCH (t: Object {class: "tree"})
            WHERE point.withinBBox(t.center, lowerLeft, upperRight)
            RETURN t as result""")

Filter for all objects of a certain type within 30 meters of given point

db.query(""" MATCH (t: Object {class: "tree"})
             WITH point.distance(point({x: -100, y:16, z:0}), t.center) as d, t
             WHERE d < 30
             RETURN t as obj, d as distance""")

Get all 2D places within 5 hops of given place

For graph-based reasoning with larger neighborhoods, there are some special built-in graph search algorithms that will be more performant that these general cypher queries. For example, for connected-component queries, you probably want to use the graph-data-science plugin

db.query(""" MATCH (p: MeshPlace {nodeSymbol: "P(1832)"})
             MATCH path=(p)-[:MESH_PLACE_CONNECTED *1..5]->(c: MeshPlace)
             RETURN DISTINCT c.nodeSymbol as ns
             ORDER BY c.nodeSymbol
         """)

Create a Room

Potentially useful for directly modifying data with LLM-generated prompt. For a more robust but less general way of adding nodes see graph_interface.py.

db.query("""WITH point({x: -95, y: 15, z: 0}) as center
            CREATE (:Room {nodeSymbol: "R(1)", center: center,  class: "test_room"})""")

Connect place near room to room

db.query("""WITH point({x: -120, y: 0, z: -100}) AS lowerLeft,
                 point({x: -70, y: 30, z:100}) AS upperRight
            MATCH (p: Place)
            MATCH (r: Room {nodeSymbol: "R(1)"})
            WHERE point.withinBBox(p.center, lowerLeft, upperRight)
            CREATE (r)-[:CONTAINS]->(p)
            """)

Query for objects in Room

Based on explicit structure of Room -> Place -> Object

db.query("""MATCH (r: Room {nodeSymbol: "R(1)"})-[:CONTAINS]->(p: Place)-[:CONTAINS]->(o: Object)
            RETURN o""")

Based on implicit / transitive structure of hierarchical relationships:

db.query("""MATCH (r: Room {nodeSymbol: "R(1)"})-[:CONTAINS*]->(o: Object)
            RETURN o""")

Reference

If you use this library, please cite us with the following:

@misc{ray2025structuredinterfaces,
      title={Structured Interfaces for Automated Reasoning with {3D} Scene Graphs},
      author={Aaron Ray and Jacob Arkin and Harel Biggie and Chuchu Fan and Luca Carlone and Nicholas Roy},
      year={2025},
      eprint={2510.16643},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2510.16643},
}