Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

添加新的数据源ES(Elasticsearch) v0.5-beta #2754

Merged
merged 33 commits into from
Aug 13, 2024

Conversation

feiazifeiazi
Copy link
Contributor

@feiazifeiazi feiazifeiazi commented Aug 8, 2024

添加Elasticsearch功能支持。(OpenSearch虽然是分支但并不支持,有差异,驱动要换,要当做新的数据源另外实现。)
@guocang

功能:

  1. 显示数据库列表。(虚拟定义的库、以__为分隔的前部分为库名。)
  2. 支持配置索引前缀,适用于小权限的用户。(通过实例管理的数据库名配置索引的前缀)
  3. 显示表列表。
  4. 显示表结构。
  5. 表数据查询。
  6. 管理查询API支持。(目前支持:/_cat/indices/*)

查询语句示例:

  1. get /_cat/indices/
  2. get /_cat/indices/dmp*?v&s=docs.count:desc
  3. GET dmp__iv/_search
  4. GET dmp__iv/_search
    {
    "query": {
    "match_all": {}
    },
    "sort": [
    {
    "CreateTime": {
    "order": "desc"
    }
    }
    ]
    ,"_source": ["DataSource", "CreateTime"]
    , "size": 5
    }

效果图:

image

image

image

image

image

image

Copy link

codecov bot commented Aug 8, 2024

Codecov Report

Attention: Patch coverage is 87.81250% with 39 lines in your changes missing coverage. Please review.

Project coverage is 77.63%. Comparing base (bd229c2) to head (d2ed4de).

Files Patch % Lines
sql/engines/elasticsearch.py 83.33% 39 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2754      +/-   ##
==========================================
+ Coverage   77.43%   77.63%   +0.19%     
==========================================
  Files         119      121       +2     
  Lines       16359    16679     +320     
==========================================
+ Hits        12668    12949     +281     
- Misses       3691     3730      +39     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@LeoQuote
Copy link
Collaborator

LeoQuote commented Aug 9, 2024

感谢贡献,关于es 支持有几个问题,可能在实现之前要仔细思考一下:

  1. 和kibana 对比,archery 有什么核心优势
  2. json的响应,如何友善展示
  3. 脱离了kibana 后如何写查询?
  4. 数据源如何管理,管理员添加集群等级的鉴权,用户自己添加索引pattern然后进行查询
  5. 授权如何管理,人员如何申请授权,如何精确控制授权

@feiazifeiazi
Copy link
Contributor Author

feiazifeiazi commented Aug 9, 2024

感谢贡献,关于es 支持有几个问题,可能在实现之前要仔细思考一下:

  1. 和kibana 对比,archery 有什么核心优势
  2. json的响应,如何友善展示
  3. 脱离了kibana 后如何写查询?
  4. 数据源如何管理,管理员添加集群等级的鉴权,用户自己添加索引pattern然后进行查询
  5. 授权如何管理,人员如何申请授权,如何精确控制授权
  1. 核心优势是archery集合了多种数据源,用一个平台可以查询。现在加多支持ES。
  2. 结果展示和mongo一样的,或说和关系型数据库一样的展示。
  3. 脱离kibana 后如何写查询, 查询是一样的写。只是没有智能提示。
  4. 数据源如何管理? 目前代码是查询所有*。能限制到用户。
  5. 精确控制授权, 和其他数据库是一样的。 选择库,选择表授权,验证权限。但是管理查询API如何限制是个问题。

@feiazifeiazi feiazifeiazi changed the title 添加新的数据源ES(Elasticsearch) 添加新的数据源ES(Elasticsearch) v0.5-beta Aug 9, 2024
@feiazifeiazi
Copy link
Contributor Author

@LeoQuote 已提交初始版,麻烦审一下。

sql/engines/elasticsearch.py Outdated Show resolved Hide resolved
sql/engines/elasticsearch.py Outdated Show resolved Hide resolved
sql/engines/elastic_search_engine_base.py Outdated Show resolved Hide resolved
sql/engines/test_elasticsearch.py Outdated Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
@LeoQuote LeoQuote merged commit 128be66 into hhyo:master Aug 13, 2024
10 checks passed
@yangguocang
Copy link

给个赞, 一直想把ES集成到archery, 没想到竟然已经实现了

@yangguocang
Copy link

哇塞 仔细一看 竟然就是我去年底在discussion 提出的支持ES的想法 感谢 @feiazifeiazi

feiazifeiazi added a commit to feiazifeiazi/Archery_Genuine2 that referenced this pull request Oct 18, 2024
---------

Co-authored-by: 王飞 <fei.wang@xgo.one>
Co-authored-by: Leo Q <LeoQuote@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants