Dev -> Main v0.0.9 (#890)

* first commit * instagram APIs implemented and AI generated caption utility added * Image upload to S3 and URL generation added * added upload to s3 instagram bucket * fixing the send email attachement issue * fixing schedule agent run * Refactored code * Resource duplicate fix (#803) * removed jpeg files used for testing * Dalle fix (#812) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * fixed recurring run issues * Supercoder Improve tool addition (#755) * fixed recurring issues * Edit agent templates (#793) * Added Controller Function * updated frontend * made minor changes in frontend * added unit test * made minor changes in frontend and backend * added another unit test * added minor change in frontend --------- Co-authored-by: anisha-contlo <anisha@contlo.com> * Readme added * instabot config folder deleted * Read file s3 fix (#823) * Updated * Read from s3 works * Read from s3 works * Minor Updates * Added Unit Tests * Updated Tests * Updated Tests * Refactored * Updated Test * Updated Test * Updated Test * refactored code * Handled the case where stable diffusion generated multiple photos * ui bug bash fixes * docker compose.yaml version reverted to original * removed the utility to add stable diffusion automatically * ui bug bash fixes * ui bug bash fixes * Edit agent templates fix (#838) * ui bug bash fixes * added test cases and modified readme * Frontend bugs (#849) * asyn function in content.js, utils refactored * minor fixes * prevent default centralisation * dropdown issue fix --------- Co-authored-by: NishantBorthakur <nist.16br@gmail.com> * Added instagram tool bucket entry in config_template.yaml * added instagram tool bucket entry in config template * ui bug fixes * ui bug fixes * ui bug fixes * External repo support for marketplace (#847) * Added support for parsing marketplace repo * Added Support for marketplace repo * Improved Marketplace logic * Fixed marketplace install * Handled no path bug * Updated .gitignore * Updated unit test * Added Unit Test * Updated .gitignore * Refactored Code * Updated Marketplace URL * Added Unit Tests * Update tool_helper.py * Update tool_helper.py --------- Co-authored-by: I’m <133493246+TransformerOptimus@users.noreply.github.com> * instagram image added * apm bug fixes * Dev rebase (#858) * Fix/agents delete (#795) * fixing the send email attachement issue * delete run label (#796) * fix * fixing schedule agent run * Updated README.md Provided Instructions for serper.dev API key creation and configuring it in the Toolkit Page * Update README.md Updated Image * Update config_template.yaml Modified Serp Configuration for SERPER * Twitter and Google Oauth for Web (#808) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * Updated DB settings (#805) * document fix (#806) * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * Bugs fixes new (#809) * csv file fix (#810) * ui bug bash fixes * ui bug bash fixes * Bugs new (#814) * Fixes (#815) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * ui bug bash fixes * ui bug bash fixes * Handling the rate limit exception * fixed add_to_vector_store_and_create_summary (#827) * fix 97971622 (#824) * Llm models fix (#829) * removed unnecessary toolkits from marketplace * Dalle fix (#825) * ui bug bash fixes (#839) * Fixing twitter creds * fixing google oauth issues * fixing google oauth issue * fixess * fixess * fixess * fixess (#854) * Resource duplicate fix (#803) * Supercoder Improve tool addition (#755) * rebased * Read file s3 fix (#823) * Updated * Read from s3 works * Read from s3 works * Minor Updates * Added Unit Tests * Updated Tests * Updated Tests * Refactored * Updated Test * Updated Test * Updated Test * Edit agent templates fix (#838) * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * rebased * first commit * instagram APIs implemented and AI generated caption utility added * Image upload to S3 and URL generation added * added upload to s3 instagram bucket * Refactored code * removed jpeg files used for testing * fixed recurring run issues * fixed recurring issues * Readme added * instabot config folder deleted * refactored code * Handled the case where stable diffusion generated multiple photos * docker compose.yaml version reverted to original * removed the utility to add stable diffusion automatically * added test cases and modified readme * Added instagram tool bucket entry in config_template.yaml * added instagram tool bucket entry in config template --------- Co-authored-by: I’m <133493246+TransformerOptimus@users.noreply.github.com> Co-authored-by: TransformerOptimus <muknrq@gmail.com> Co-authored-by: Phoenix2809 <133874957+Phoenix2809@users.noreply.github.com> Co-authored-by: Taran <97586318+Tarraann@users.noreply.github.com> Co-authored-by: Tarraann <jot.taran15522@gmail.com> Co-authored-by: luciferlinx <129729795+luciferlinx101@users.noreply.github.com> Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> Co-authored-by: Kalki <parth.pathak@science.christuniversity.in> Co-authored-by: Kalki <97698934+jedan2506@users.noreply.github.com> Co-authored-by: GeekyBaller <43145646+neelayan7@users.noreply.github.com> Co-authored-by: Nishant Borthakur <101320057+nborthy@users.noreply.github.com> Co-authored-by: NishantBorthakur <nist.16br@gmail.com> Co-authored-by: Autocop-Agent <129729746+Autocop-Agent@users.noreply.github.com> Co-authored-by: Anisha Gupta <60440541+anisha1607@users.noreply.github.com> Co-authored-by: Maverick-F35 <maverickcode.f35@gmail.com> * apm bug fixes * Docker image (#836) * handled resource path change (#861) * handled resource path change * readme changed * added new test case for stable diffusion * test cases refactored * Updated Expiry Time backward fix * readme toolkit fix (#867) * Knowledge frontend (#860) * knowledge first cut * knowledge details * knowledge add form * agent create knowledge select * add database part 1 * add database part 2 * local storage fixes * database details * minor fix * comments resolution * knowledge details dropdown * knowledge form added * delete database * delete database modal * delete database modal 2 * minorf ix * marketplace for knowledge * update database changes * dropdown changes * uninstall knowledge in marketplace * minor fixes * minor fixws * api integration * bug fixes on db connect * bug fix on deleting database * knowledge dropdown fi * model fixes * minor fiox * remove tab fixes * minor fix * loading text fixes * index validation * uninstall knowledge api * minor fix * minor time fix * code refactoring * knowledge id changed to knowledge name * introduction and use cases added * uninstall marketplace template by id * minor fixes * Changes in API * Changea * bug fixes where adding loader while installing, icon alignment, selecting a knowledge made compulsory, knowledge sidebar issues are fixed, adding of loader in marketplace, connection of database, uninstalling knowledge resolved, dropdown bug by mayur resolved, knowledge id was not going with the agent configs in, resolved it * minor bug fixes * adding of installed state in the cards of knowledge marketplace * install template condtion * merging with dev of missed part * install dropdown bug fix * select index validation * delete db prompt change * time issue fix * connect db response handling * minor fix * final changes * minor fix --------- Co-authored-by: NishantBorthakur <nist.16br@gmail.com> Co-authored-by: Tarraann <jot.taran15522@gmail.com> Co-authored-by: namansleeps <mandhani12@gmail.com> * Fixed Delete Agent Bug (#830) * made the fix * changed deleted status to stopped * close button fix * refactor * centralise prevent default * app.css fixes * Dev fixes (#869) * close button fix * refactor * centralise prevent default * app.css fixes --------- Co-authored-by: NishantBorthakur <nist.16br@gmail.com> * close icon issue * Added tools.json (#868) * Updated Chat completion for tool selection prompt * Updated Prompt * Bug fixes afternoon (#875) * scheduling bug fixes * agent templates install bug fix * Docker image (#876) * temp commit * use config.yaml in docker-compose.yaml * temp commit * temp commit * Fix/agents delete (#795) * fixing the send email attachement issue * delete run label (#796) * fix * works with slim * fixing schedule agent run * Updated README.md Provided Instructions for serper.dev API key creation and configuring it in the Toolkit Page * optimize gui build * only copy necessary file * images build * Update README.md Updated Image * Update config_template.yaml Modified Serp Configuration for SERPER * Twitter and Google Oauth for Web (#808) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * Updated DB settings (#805) * document fix (#806) * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * ui bug bash fixes * Bugs fixes new (#809) * csv file fix (#810) * ui bug bash fixes * build nginx * ui bug bash fixes * Bugs new (#814) * Fixes (#815) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * ui bug bash fixes * ui bug bash fixes * Handling the rate limit exception * fixed add_to_vector_store_and_create_summary (#827) * fix 97971622 (#824) * Llm models fix (#829) * removed unnecessary toolkits from marketplace * revert nginx change * docker gui for dev * docker ignore update * docker ignore update for gui * Dalle fix (#825) * bumpupto 3.10 * ui bug bash fixes (#839) * Fixing twitter creds * fixing google oauth issues * fixing google oauth issue * remove git from dependencies * remove --only=production * install tool dependencies on reboot * add docker compose example to use docker images --------- Co-authored-by: I’m <133493246+TransformerOptimus@users.noreply.github.com> Co-authored-by: BoundlessAsura <122777244+boundless-asura@users.noreply.github.com> Co-authored-by: TransformerOptimus <muknrq@gmail.com> Co-authored-by: Boundless Asura <nihir@contlo.com> Co-authored-by: Phoenix2809 <133874957+Phoenix2809@users.noreply.github.com> Co-authored-by: Taran <97586318+Tarraann@users.noreply.github.com> Co-authored-by: Tarraann <jot.taran15522@gmail.com> Co-authored-by: luciferlinx <129729795+luciferlinx101@users.noreply.github.com> Co-authored-by: Kalki <parth.pathak@science.christuniversity.in> Co-authored-by: Kalki <97698934+jedan2506@users.noreply.github.com> Co-authored-by: GeekyBaller <43145646+neelayan7@users.noreply.github.com> Co-authored-by: Nishant Borthakur <101320057+nborthy@users.noreply.github.com> Co-authored-by: NishantBorthakur <nist.16br@gmail.com> * Final knowledge backend (#870) * very minor fix for naming * Changes for HTTP EXception in Knowledge Controller * HTTP changes for vector_dbs * Bug fixes afternoon (#878) * scheduling bug fixes * agent templates install bug fix * tool templates fixes * Refactoring of vector embedding * adding of knowledge logo * Delete HTTP fixed * Updated prompt * changes in tests * changes * changes * args fix in dalle * bug fix * minor change * minor changes * fixes (#886) * the user will now be unable to select striked knowledges * minor fix * Fix for agent details - not fetching data * minor change for details * Fixes --------- Co-authored-by: Maverick-F35 <maverickcode.f35@gmail.com> Co-authored-by: TransformerOptimus <muknrq@gmail.com> Co-authored-by: Taran <97586318+Tarraann@users.noreply.github.com> Co-authored-by: Tarraann <jot.taran15522@gmail.com> Co-authored-by: Autocop-Agent <129729746+Autocop-Agent@users.noreply.github.com> Co-authored-by: Anisha Gupta <60440541+anisha1607@users.noreply.github.com> Co-authored-by: anisha-contlo <anisha@contlo.com> Co-authored-by: Kalki <parth.pathak@science.christuniversity.in> Co-authored-by: Nishant Borthakur <101320057+nborthy@users.noreply.github.com> Co-authored-by: NishantBorthakur <nist.16br@gmail.com> Co-authored-by: Maverick-F35 <138012351+Maverick-F35@users.noreply.github.com> Co-authored-by: I’m <133493246+TransformerOptimus@users.noreply.github.com> Co-authored-by: BoundlessAsura <122777244+boundless-asura@users.noreply.github.com> Co-authored-by: Phoenix2809 <133874957+Phoenix2809@users.noreply.github.com> Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> Co-authored-by: Kalki <97698934+jedan2506@users.noreply.github.com> Co-authored-by: GeekyBaller <43145646+neelayan7@users.noreply.github.com> Co-authored-by: namansleeps <mandhani12@gmail.com> Co-authored-by: Boundless Asura <nihir@contlo.com> Co-authored-by: namansleeps <122260931+namansleeps@users.noreply.github.com>
TransformerOptimus · Jul 27, 2023 · a7f8fb8 · a7f8fb8
1 parent eeac9cc
commit a7f8fb8
Show file tree

Hide file tree

Showing 137 changed files with 6,253 additions and 721 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -1,2 +1,18 @@
-venv/
-gui/
+# Ignore everything
+**
+
+# Allow files and directories
+!/migrations
+!/nginx
+!/superagi
+!/tgwui
+!/tools
+!/workspace
+!/main.py
+!/requirements.txt
+!/entrypoint.sh
+!/entrypoint_celery.sh
+!/wait-for-it.sh
+!/tools.json
+!/install_tool_dependencies.sh
+!/alembic.ini
diff --git a/.gitignore b/.gitignore
@@ -17,3 +17,7 @@ workspace/output
 workspace/input
 celerybeat-schedule
 ../bfg-report*
+superagi/tools/marketplace_tools/
+superagi/tools/external_tools/
+tests/unit_tests/resource_manager/test_path
+tools.json
diff --git a/Dockerfile b/Dockerfile
@@ -1,15 +1,33 @@
-FROM python:3.9
+# Stage 1: Compile image
+FROM python:3.10-slim-bullseye AS compile-image
 WORKDIR /app
-COPY requirements.txt .
 
-#RUN apt-get update && apt-get install --no-install-recommends -y git wget libpq-dev gcc python3-dev && pip install psycopg2
-RUN pip install --upgrade pip
-RUN pip install --no-cache-dir -r requirements.txt
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y wget libpq-dev gcc g++ python3-dev && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN python -m venv /opt/venv
+ENV PATH="/opt/venv/bin:$PATH"
+
+COPY requirements.txt .
+RUN pip install --upgrade pip && \
+    pip install --no-cache-dir -r requirements.txt
 
 COPY . .
-COPY config.yaml ./config.yaml
-COPY entrypoint.sh ./entrypoint.sh
-COPY wait-for-it.sh ./wait-for-it.sh
-RUN chmod +x ./entrypoint.sh ./wait-for-it.sh
 
-CMD ["./wait-for-it.sh", "super__postgres:5432","-t","60","--","./entrypoint.sh"]
+RUN chmod +x ./entrypoint.sh ./wait-for-it.sh ./install_tool_dependencies.sh ./entrypoint_celery.sh
+
+# Stage 2: Build image
+FROM python:3.10-slim-bullseye AS build-image
+WORKDIR /app
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends -y libpq-dev && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY --from=compile-image /opt/venv /opt/venv
+COPY --from=compile-image /app /app
+
+ENV PATH="/opt/venv/bin:$PATH"
diff --git a/DockerfileCelery b/DockerfileCelery
@@ -8,9 +8,16 @@ RUN pip install --upgrade pip
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
 
-WORKDIR /app
 COPY . .
-COPY config.yaml .
+
+# Downloads the tools
+RUN python superagi/tool_manager.py
+
+# Set executable permissions for install_tool_dependencies.sh
+RUN chmod +x install_tool_dependencies.sh
+
+# Install dependencies
+RUN ./install_tool_dependencies.sh
 
 # Downloads the tools
 RUN python superagi/tool_manager.py

diff --git a/config_template.yaml b/config_template.yaml
@@ -38,6 +38,7 @@ RESOURCES_OUTPUT_ROOT_DIR: workspace/output/{agent_id}/{agent_execution_id} # Fo
 
 #S3 RELATED DETAILS ONLY WHEN STORAGE_TYPE IS "S3"
 BUCKET_NAME:
+INSTAGRAM_TOOL_BUCKET_NAME:                                   #Public read bucket, Images generated by stable diffusion are put in this bucket and the public url of the same is generated.
 AWS_ACCESS_KEY_ID:
 AWS_SECRET_ACCESS_KEY:
 

diff --git a/docker-compose.image.example.yaml b/docker-compose.image.example.yaml
@@ -0,0 +1,75 @@
+version: '3.8'
+services:
+  backend:
+    image: "superagidev/backend:dev"
+    depends_on:
+      - super__redis
+      - super__postgres
+    networks:
+      - super_network
+    env_file:
+      - config.yaml
+    command: ["/app/wait-for-it.sh", "super__postgres:5432","-t","60","--","/app/entrypoint.sh"]
+
+  celery:
+    image: "superagidev/backend:dev"
+    depends_on:
+      - super__redis
+      - super__postgres
+    networks:
+      - super_network
+    env_file:
+      - config.yaml
+    command: ["/app/entrypoint_celery.sh"]
+    volumes:
+      - "./workspace:/app/workspace"
+
+  gui:
+    image: "superagidev/gui:dev"
+    environment:
+      - NEXT_PUBLIC_API_BASE_URL=/api
+    networks:
+      - super_network
+
+  super__redis:
+    image: "redis/redis-stack-server:latest"
+    networks:
+      - super_network
+#    uncomment to expose redis port to host
+#    ports:
+#      - "6379:6379"
+    volumes:
+      - redis_data:/data
+
+  super__postgres:
+    image: "docker.io/library/postgres:latest"
+    environment:
+      - POSTGRES_USER=superagi
+      - POSTGRES_PASSWORD=password
+      - POSTGRES_DB=super_agi_main
+    volumes:
+      - superagi_postgres_data:/var/lib/postgresql/data/
+    networks:
+      - super_network
+#    uncomment to expose postgres port to host
+#    ports:
+#      - "5432:5432"
+
+  proxy:
+    image: nginx:stable-alpine
+    ports:
+      - "3000:80"
+    networks:
+      - super_network
+    depends_on:
+      - backend
+      - gui
+    volumes:
+      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf
+
+networks:
+  super_network:
+    driver: bridge
+volumes:
+  superagi_postgres_data:
+  redis_data:
diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -9,22 +9,23 @@ services:
       - super__postgres
     networks:
       - super_network
+    command: ["/app/wait-for-it.sh", "super__postgres:5432","-t","60","--","/app/entrypoint.sh"]
   celery:
     volumes:
       - "./:/app"
       - "${EXTERNAL_RESOURCE_DIR:-./workspace}:/app/ext"
-    build:
-      context: .
-      dockerfile: DockerfileCelery
+    build: .
     depends_on:
       - super__redis
       - super__postgres
     networks:
       - super_network
+    command: ["/app/entrypoint_celery.sh"]
   gui:
-    build: ./gui
-    environment:
-      - NEXT_PUBLIC_API_BASE_URL=/api
+    build:
+      context: ./gui
+      args:
+        NEXT_PUBLIC_API_BASE_URL: "/api"
     networks:
       - super_network
 #    volumes:

diff --git a/entrypoint.sh b/entrypoint.sh
@@ -1,11 +1,8 @@
 #!/bin/bash
 
-# Downloads the tools
+# Downloads the tools from marketplace and external tool repositories
 python superagi/tool_manager.py
 
-# Set executable permissions for install_tool_dependencies.sh
-chmod +x install_tool_dependencies.sh
-
 # Install dependencies
 ./install_tool_dependencies.sh
 

diff --git a/entrypoint_celery.sh b/entrypoint_celery.sh
@@ -1,5 +1,9 @@
 #!/bin/bash
-Xvfb :0 -screen 0 1280x1024x24 &
-x11vnc -display :0 -N -forever -shared &
 
-exec "$@"
+# Downloads the tools
+python superagi/tool_manager.py
+
+# Install dependencies
+./install_tool_dependencies.sh
+
+exec celery -A superagi.worker worker --beat --loglevel=info
diff --git a/gui/.dockerignore b/gui/.dockerignore
@@ -1,2 +1,13 @@
-node_modules/
-.next/
+# Ignore everything
+**
+
+# Allow files and directories
+!app
+!pages
+!public
+!utils
+!package.json
+!next.config.js
+!package-lock.json
+!.eslintrc.json
+!jsconfig.json
diff --git a/gui/Dockerfile b/gui/Dockerfile
@@ -1,13 +1,19 @@
-FROM node:lts AS deps
-
+FROM node:18-alpine AS deps
+RUN apk add --no-cache libc6-compat
 WORKDIR /app
-
-COPY package*.json ./
+COPY package.json package-lock.json ./
 RUN npm ci
 
-FROM node:lts AS builder
+# Rebuild the source code only when needed
+FROM node:18-alpine AS builder
+
 WORKDIR /app
-COPY . .
+
 COPY --from=deps /app/node_modules ./node_modules
 
+COPY . .
+ARG NEXT_PUBLIC_API_BASE_URL=/api
+ENV NEXT_PUBLIC_API_BASE_URL=$NEXT_PUBLIC_API_BASE_URL
+EXPOSE 3000
+
 CMD ["npm", "run", "dev"]
diff --git a/gui/DockerfileProd b/gui/DockerfileProd
@@ -1,11 +1,43 @@
-FROM node:lts
+FROM node:18-alpine AS deps
+RUN apk add --no-cache libc6-compat
+WORKDIR /app
+COPY package.json package-lock.json ./
+RUN npm ci --only=production
+
+# Rebuild the source code only when needed
+FROM node:18-alpine AS builder
 
 WORKDIR /app
 
-COPY package.json .
-RUN npm install
+COPY --from=deps /app/node_modules ./node_modules
 
 COPY . .
+ARG NEXT_PUBLIC_API_BASE_URL=/api
+ENV NEXT_PUBLIC_API_BASE_URL=$NEXT_PUBLIC_API_BASE_URL
 
 RUN npm run build
-CMD ["npm", "run", "start"]
+
+# Production image, copy all the files and run next
+FROM node:18-alpine AS runner
+WORKDIR /app
+
+ENV NODE_ENV production
+
+RUN addgroup --system --gid 1001 supergroup
+RUN adduser --system --uid 1001 superuser
+
+COPY --from=builder /app/public ./public
+COPY --from=builder /app/package.json ./package.json
+
+# Automatically leverage output traces to reduce image size
+# https://nextjs.org/docs/advanced-features/output-file-tracing
+COPY --from=builder --chown=superuser:supergroup /app/.next/standalone ./
+COPY --from=builder --chown=superuser:supergroup /app/.next/static ./.next/static
+
+USER superuser
+
+EXPOSE 3000
+
+ENV PORT 3000
+
+CMD ["node", "server.js"]
diff --git a/gui/next.config.js b/gui/next.config.js
@@ -1,6 +1,7 @@
 /** @type {import('next').NextConfig} */
 const nextConfig = {
   assetPrefix: process.env.NODE_ENV === "production" ? "/" : "./",
+  output: 'standalone'
 };
 
 module.exports = nextConfig;
diff --git a/gui/package-lock.json b/gui/package-lock.json
diff --git a/gui/package.json b/gui/package.json
@@ -29,6 +29,7 @@
     "react-draggable": "^4.4.5",
     "react-grid-layout": "^1.3.4",
     "react-markdown": "^8.0.7",
+    "react-spinners": "^0.13.8",
     "react-toastify": "^9.1.3"
   }
 }