RC4 rebase master (vesoft-inc#2034)

* Upgrade rocksdb to 6.7.3 (vesoft-inc#1948) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: yaphet <darion.wang@vesoft.com> * Detect OS type and exit if not built in Linux (vesoft-inc#1951) * Detect OS type and exit if not built in Linux Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> * Disable for FreeBSD Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> * [doc] user-permission-mng (vesoft-inc#1929) * alter user & change psw * built in roles * cbs comments * fix comments * update table * addenable_authorize doc * [doc] ttl doc modification (vesoft-inc#1939) * ttl doc modification * fix comments * yt comments * silver days comments * timestamp update * Let action not fail fast. (vesoft-inc#1909) * Let action not fail fast. * Disable the ci will product artifact. * Remove removePrefix interface in kvstore (vesoft-inc#1953) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: yaphet <darion.wang@vesoft.com> * using admin client in MetaServiceHandler (vesoft-inc#1864) * [doc] fix typo (vesoft-inc#1954) * [doc]show statements (vesoft-inc#1924) * [doc]customized dir (vesoft-inc#1906) * Fixed a bug caused by not filtering the versions of edges (vesoft-inc#1962) * Fixed a bug caused by not filtering the versions of edges when executing "go from vid over *". * address CPWstatic's comment. * Modify the listEdges interface and add parameters to directly return all latest versions of edges. * address laura's comment. Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * [doc]modify job manger doc (vesoft-inc#1957) * modify job manger doc * add flush Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Fix delete verticse (vesoft-inc#1970) * Fix the read performance issue after delete range with rocksdb-5.15.10 (vesoft-inc#1973) * Fix the read performace issue after delete range with rocksdb-5.15.10 * Fix the move issue * fix the move issue * Rebase and address bright-starry-sky's comments Co-authored-by: heng <heng.chen@vesoft.com> * Remove the LOG(FATAL) when encounting the unknown log type (vesoft-inc#1978) * 1,Improve customer friendliness; 2, fixed typo error (vesoft-inc#1980) * upgrade spark download address (vesoft-inc#1981) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * fix-typo-in-doc (vesoft-inc#1976) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * modify commands in build doc (vesoft-inc#1969) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * modify log level (vesoft-inc#1945) Co-authored-by: dangleptr <37216992+dangleptr@users.noreply.github.com> Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * use bloom filter, add event listener to collect stats of compaction, improve update rocksdb options (vesoft-inc#1959) * open bloom filter in default conf, add event listener to collect stats of compaction * imporve update rocksdb configs * fix ut error Co-authored-by: yaphet <darion.wang@vesoft.com> * Update the interface atomicOp to support empty string (vesoft-inc#2002) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * update to rc4 (vesoft-inc#1968) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Fixed if you execute "go from xx over *", and when there is no edge under a space, (vesoft-inc#1983) an unreasonable error message is returned (SpaceId `xxx 'is nonexistent) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * [doc]pr-commit-msg (vesoft-inc#1993) * pr-commit-msg * add examples Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * doc:modify ttl query (vesoft-inc#2006) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Let merge coverage report faster by fastcov instead of lcov. (vesoft-inc#1943) Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Auto check keyword (vesoft-inc#1922) * Supports the detection keyword, name_label supports string * add tests * address Shylock's comment * modify to add reserved keyword Co-authored-by: dutor <440396+dutor@users.noreply.github.com> * Fix coredump for update if encountering bad format row (vesoft-inc#2024) Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com> * fixed loop depend problem for compile link (vesoft-inc#1995) * fixed loop depend problem * fixed error for clang9 and gcc9 * Addressed laura-ding's comment * Check all reader is null or not before using it (vesoft-inc#2029) * fix go without yield return errror result (vesoft-inc#2012) * fix go without yield return errror result * fix format * address dangleptr's comment * address issue change TTL_DURATION not negative * add onlyStructure is true test case Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com> * fixed bug for delete vertex without edge (vesoft-inc#2001) * When deleting a vertex, if the edge does not exist, then we should continue to delete the vertex instead of reporting an error. * added the testcase for delete vertex without edge. * add alias for lookup (vesoft-inc#2000) * fix bug 1990/1991 (vesoft-inc#2007) * 1. meta client sub mit job only to leader 2. sort jobs in desc id order * now show jobs return in desc order, adjust UT Co-authored-by: dangleptr <37216992+dangleptr@users.noreply.github.com> * upload package to oss (vesoft-inc#1972) * upload oss * update * Remove redundant * address @yixinlu's comment Co-authored-by: dutor <440396+dutor@users.noreply.github.com> Co-authored-by: dangleptr <37216992+dangleptr@users.noreply.github.com> Co-authored-by: heng <heng.chen@vesoft.com> Co-authored-by: yaphet <darion.wang@vesoft.com> Co-authored-by: JaySon <jayson.hjs@gmail.com> Co-authored-by: Amber Zhang <42762957+Amber1990Zhang@users.noreply.github.com> Co-authored-by: Shylock Hg <33566796+Shylock-Hg@users.noreply.github.com> Co-authored-by: Simon Liu <331435+monadbobo@users.noreply.github.com> Co-authored-by: dutor <440396+dutor@users.noreply.github.com> Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com> Co-authored-by: Liuxue-Yang <50105172+Liuxue-Yang@users.noreply.github.com> Co-authored-by: laura-ding <48548375+laura-ding@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: liuyu85cn <52276794+liuyu85cn@users.noreply.github.com>
jaemybai · Apr 1, 2020 · 2ea95f3 · 2ea95f3
1 parent 50b2af1
commit 2ea95f3
Show file tree

Hide file tree

Showing 187 changed files with 2,189 additions and 651 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -13,6 +13,7 @@ jobs:
     name: build
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
         os:
           - centos6
@@ -58,7 +59,12 @@ jobs:
             -DCMAKE_C_COMPILER=clang \
             -DENABLE_ASAN=on \
             -B build/
+      - name: Make with one thread
+        if: matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7'
+        run: |
+          cmake --build build/
       - name: Make
+        if: matrix.compiler != 'gcc-9.2' || matrix.os != 'centos7'
         run: |
           cmake --build build/ -j $(nproc)
       - name: CTest with multiple thread
@@ -90,10 +96,8 @@ jobs:
         if: success() && matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7'
         run: |
           set -e
-          /usr/local/bin/lcov --version
-          /usr/local/bin/lcov --capture --gcov-tool $GCOV --directory . --output-file coverage.info
-          /usr/local/bin/lcov --remove coverage.info '*/opt/vesoft/*' -o clean.info
-          bash <(curl -s https://codecov.io/bash) -Z -f clean.info
+          fastcov -o coverage.info -g $GCOV -l --exclude=/opt/vesoft/ --exclude=scanner.lex
+          bash <(curl -s https://codecov.io/bash) -Z -f coverage.info
         env:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
         shell: bash
diff --git a/.github/workflows/package.yaml b/.github/workflows/package.yaml
@@ -4,6 +4,7 @@ on:
   schedule:
     - cron: '0 18 * * *'
   push:
+    # Releases are all branch based, and the branch package here is for testing.
     branches:
       - 'v[0-9]+.*'
     paths-ignore:
@@ -30,10 +31,20 @@ jobs:
         shell: bash
       - name: Compute version
         id: version
+        env:
+          CPACK_OUTPUT_DIR: build/cpack_output
         run: |
           [[ "${{ github.event_name }}" == "push" ]] && \
             echo "::set-output name=version::$(cut -d/ -f3 <<< ${{ github.ref }})" || \
-            echo "::set-output name=version::nightly"
+            (echo "::set-output name=version::nightly"
+              cd $CPACK_OUTPUT_DIR && \
+              filename=$(find . -type f \( -iname \*.deb -o -iname \*.rpm \) -exec basename {} \;)
+              echo "::set-output name=filepath::$CPACK_OUTPUT_DIR/$filename")
+        shell: bash
+      - name: upload oss
+        if: github.event_name != 'push'
+        run: |
+          ./ci/scripts/upload-oss.sh OSS_ENDPOINT=${{ secrets.OSS_ENDPOINT }} OSS_ID=${{ secrets.OSS_ID }} OSS_SECRET=${{ secrets.OSS_SECRET }} filepath=${{ steps.version.outputs.filepath }} nightly=true
         shell: bash
       - uses: actions/upload-artifact@v1
         with:

diff --git a/.github/workflows/pull_request.yaml b/.github/workflows/pull_request.yaml
@@ -26,6 +26,7 @@ jobs:
     needs: lint
     runs-on: self-hosted
     strategy:
+      fail-fast: false
       matrix:
         os:
           - centos7
@@ -110,10 +111,8 @@ jobs:
         if: success() && matrix.compiler == 'gcc-9.2' && matrix.os == 'centos7' && steps.ignore_docs.outputs.num_source_files != 0
         run: |
           set -e
-          /usr/local/bin/lcov --version
-          /usr/local/bin/lcov --capture --gcov-tool $GCOV --directory . --output-file coverage.info
-          /usr/local/bin/lcov --remove coverage.info '*/opt/vesoft/*' -o clean.info
-          bash <(curl -s https://codecov.io/bash) -Z -t $(cat /etc/action/secrets/codecov) -f clean.info
+          fastcov -o coverage.info -l --exclude=/opt/vesoft/ --exclude=scanner.lex
+          bash <(curl -s https://codecov.io/bash) -Z -t $(cat /etc/action/secrets/codecov) -f coverage.info
         shell: bash
       - name: Cleanup
         if: always()

diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -40,12 +40,9 @@ jobs:
         run: |
           ./ci/scripts/upload-github-release-asset.sh github_token=${{ secrets.GITHUB_TOKEN }} repo=${{ github.repository }} tag=${{ steps.vars.outputs.tag }} filepath=${{ steps.vars.outputs.filepath }}
           ./ci/scripts/upload-github-release-asset.sh github_token=${{ secrets.GITHUB_TOKEN }} repo=${{ github.repository }} tag=${{ steps.vars.outputs.tag }} filepath=${{ steps.vars.outputs.shafilepath }}
+          ./ci/scripts/upload-oss.sh OSS_ENDPOINT=${{ secrets.OSS_ENDPOINT }} OSS_ID=${{ secrets.OSS_ID }} OSS_SECRET=${{ secrets.OSS_SECRET }} tag=${{ steps.vars.outputs.tag }} filepath=${{ steps.vars.outputs.filepath }}
+          ./ci/scripts/upload-oss.sh OSS_ENDPOINT=${{ secrets.OSS_ENDPOINT }} OSS_ID=${{ secrets.OSS_ID }} OSS_SECRET=${{ secrets.OSS_SECRET }} tag=${{ steps.vars.outputs.tag }} filepath=${{ steps.vars.outputs.shafilepath }}
         shell: bash
-      - name: upload release artifacts
-        uses: actions/upload-artifact@v1
-        with:
-          name: ${{ matrix.os }}-release
-          path: build/cpack_output
 
   docker:
     name: build docker images

diff --git a/.linters/cpp/checkKeyword.py b/.linters/cpp/checkKeyword.py
@@ -0,0 +1,139 @@
+# signout--coding:utf-8--
+
+# Copyright (c) 2019 vesoft inc. All rights reserved.
+#
+# This source code is licensed under Apache 2.0 License,
+# attached with Common Clause Condition 1.0, found in the LICENSES directory.
+
+import os
+import sys
+import re
+
+
+PASER_FILE_PATH = 'src/parser/parser.yy'
+SCANNER_FILE_PATH = 'src/parser/scanner.lex'
+
+reserved_key_words = [
+    'KW_GO',
+    'KW_AS',
+    'KW_TO',
+    'KW_OR',
+    'KW_AND',
+    'KW_XOR',
+    'KW_USE',
+    'KW_SET',
+    'KW_FROM',
+    'KW_WHERE',
+    'KW_MATCH',
+    'KW_INSERT',
+    'KW_YIELD',
+    'KW_RETURN',
+    'KW_DESCRIBE',
+    'KW_DESC',
+    'KW_VERTEX',
+    'KW_EDGE',
+    'KW_EDGES',
+    'KW_UPDATE',
+    'KW_UPSERT',
+    'KW_WHEN',
+    'KW_DELETE',
+    'KW_FIND',
+    'KW_LOOKUP',
+    'KW_ALTER',
+    'KW_STEPS',
+    'KW_OVER',
+    'KW_UPTO',
+    'KW_REVERSELY',
+    'KW_INDEX',
+    'KW_INDEXES',
+    'KW_REBUILD',
+    'KW_INT',
+    'KW_BIGINT',
+    'KW_DOUBLE',
+    'KW_STRING',
+    'KW_BOOL',
+    'KW_TIMESTAMP',
+    'KW_TAG',
+    'KW_TAGS',
+    'KW_UNION',
+    'KW_INTERSECT',
+    'KW_MINUS',
+    'KW_NO',
+    'KW_OVERWRITE',
+    'KW_SHOW',
+    'KW_ADD',
+    'KW_CREATE',
+    'KW_DROP',
+    'KW_REMOVE',
+    'KW_IF',
+    'KW_NOT',
+    'KW_EXISTS',
+    'KW_WITH',
+    'KW_CHANGE',
+    'KW_GRANT',
+    'KW_REVOKE',
+    'KW_ON',
+    'KW_BY',
+    'KW_IN',
+    'KW_DOWNLOAD',
+    'KW_GET',
+    'KW_OF',
+    'KW_ORDER',
+    'KW_INGEST',
+    'KW_COMPACT',
+    'KW_FLUSH',
+    'KW_SUBMIT',
+    'KW_ASC',
+    'KW_DISTINCT',
+    'KW_FETCH',
+    'KW_PROP',
+    'KW_BALANCE',
+    'KW_STOP',
+    'KW_LIMIT',
+    'KW_OFFSET',
+    'KW_GROUP',
+    'KW_IS',
+    'KW_NULL',
+    'KW_FORCE',
+    'KW_RECOVER'
+]
+
+
+def get_unreserved_keyword(file_path):
+    parser_file = open(file_path)
+    flag = 0
+    unreserved_key_words = []
+    for line in parser_file.readlines():
+        if line.strip() == 'unreserved_keyword':
+            flag = 1
+            continue
+        if flag == 1:
+            if line.strip() == ';':
+                break
+            unreserved_key_words.append(re.sub('\\s+[:|]\\s+(\\w+)\\s+.*', '\\1', line).strip())
+            continue
+
+    parser_file.close()
+    return unreserved_key_words
+
+
+if __name__ == '__main__':
+    cmd = 'git diff --diff-filter=ACMRTUXB HEAD -p ' + SCANNER_FILE_PATH + '|grep "^+"|grep -v "^+++"|grep "KW_"'
+    content = os.popen(cmd)
+    keywords=[]
+    for line in  content.readlines():
+        keyword = re.sub('.*(KW_\\w+)\s*;.*','\\1',line.strip())
+        keywords.append(keyword)
+
+    if len(keywords) == 0:
+        exit(0)
+    unreserved_key_words = get_unreserved_keyword(PASER_FILE_PATH)
+    new_key_words = [word for word in keywords if word not in reserved_key_words]
+    if len(new_key_words) == 0:
+        exit(0)
+    result = [word for word in new_key_words if word not in unreserved_key_words]
+    if len(result) == 0:
+        exit(0)
+    print('Keywords \"{}\" in src/parser/scanner.lex are not in the unreserved keyword list.'.format(result))
+    print('Please add them to the unreserved keyword in the src/parser/parser.yy.')
+    exit(1)
diff --git a/.linters/cpp/hooks/pre-commit.sh b/.linters/cpp/hooks/pre-commit.sh
@@ -6,6 +6,15 @@
 # attached with Common Clause Condition 1.0, found in the LICENSES directory.
 
 CPPLINT=`dirname $0`/../../.linters/cpp/cpplint.py
+CHECKKEYWORD=`dirname $0`/../../.linters/cpp/checkKeyword.py
+
+echo "Performing checkout keyword..."
+python $CHECKKEYWORD
+
+if [ $? -ne 0 ]; then
+    echo "Checkout keyword failed"
+    exit 1
+fi
 
 if [ $# -eq 0 ];then
     # Since cpplint.py could only apply on our working tree,

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -30,6 +30,11 @@ project("Nebula Graph" C CXX)
 
 # Set the project home dir
 set(NEBULA_HOME ${CMAKE_CURRENT_SOURCE_DIR})
+
+if(${CMAKE_SYSTEM_NAME} MATCHES "(Darwin|FreeBSD|Windows)")
+	MESSAGE(FATAL_ERROR "Only Linux is supported")
+endif ()
+
 # To include customized FindXXX.cmake modules
 set(CMAKE_MODULE_PATH "${NEBULA_HOME}/cmake" ${CMAKE_MODULE_PATH})
 include(LinkerConfig)

diff --git a/ci/scripts/upload-oss.sh b/ci/scripts/upload-oss.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+#
+# This script  upload package to oss.
+#
+# - OSS_ENDPOINT
+# - OSS_ID
+# - OSS_SECRET
+# - filepath
+# - tag
+# - nightly
+#
+# Example:
+#
+#   upload-oss.sh OSS_ENDPOINT=xxx OSS_ID=xxx OSS_SECRET=xxx tag=v0.1.0 filepath=xxx
+#   upload-oss.sh OSS_ENDPOINT=xxx OSS_ID=xxx OSS_SECRET=xxx filepath=xxx nightly=true
+
+set -e
+
+for op in $@; do
+  eval "$op"
+done
+
+if [[ $nightly != "" ]]; then
+  OSS_SUBDIR=`date +%Y%m%d`
+  OSS_URL="oss://nebula-graph/build-deb"/nightly/${OSS_SUBDIR}
+else
+  OSS_SUBDIR=`echo $tag |sed 's/^v//'`
+  OSS_URL="oss://nebula-graph/build-deb"/${OSS_SUBDIR}
+fi
+
+echo "Uploading oss... "
+ossutil64 -e ${OSS_ENDPOINT} -i ${OSS_ID} -k ${OSS_SECRET} -f cp ${filepath} ${OSS_URL}/$(basename ${filepath})
diff --git a/cmake/LinkerConfig.cmake b/cmake/LinkerConfig.cmake
@@ -9,7 +9,7 @@ execute_process(
     OUTPUT_STRIP_TRAILING_WHITESPACE
 )
 
-if (${default_linker_type} STREQUAL "ld")
+if ("${default_linker_type}" STREQUAL "ld")
     set(default_linker_type "bfd")
 endif()