Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

修复mongodb不能执行长sql的问题 #1556

Merged
merged 5 commits into from
Jun 3, 2022
Merged

修复mongodb不能执行长sql的问题 #1556

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5272983
修复mongodb不能执行长sql的bug
czxin788 May 30, 2022
876110d
临时文件用tempfile.NamedTemporaryFile()方法，使代码更优雅
czxin788 May 30, 2022
3d31f13
移除logger.debug(cmd)，为过代码扫描
czxin788 May 30, 2022
70dfa35
修复mongodb不显示执行结果的问题
czxin788 May 31, 2022
c9f16f5
当mongodb sql长度大于4000个字符时，使用load方法执行sql；小于4000个字符时，直接执行sql，这样可以减少磁盘交换，…
czxin788 Jun 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions sql/engines/mongo.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
import subprocess
import simplejson as json
import datetime
import uuid
import os
from bson.son import SON
from bson import json_util
from pymongo.errors import OperationFailure
Expand Down Expand Up @@ -254,14 +256,21 @@ class MongoEngine(EngineBase):

def exec_cmd(self, sql, db_name=None, slave_ok=''):
"""审核时执行的语句"""


if self.user and self.password and self.port and self.host:
msg = ""
auth_db = self.instance.db_name or 'admin'
# 以uuid命名临时mongo脚本文件，防止冲突
uuid_ = str(uuid.uuid4())
tempfile = '/opt/archery/scripts/mongodb/' + uuid_ + '.js'
LeoQuote marked this conversation as resolved.
Show resolved Hide resolved
with open(f'{tempfile}', 'w') as f:
f.write(sql)
try:
if not sql.startswith('var host='): #在master节点执行的情况
cmd = "{mongo} --quiet -u {uname} -p '{password}' {host}:{port}/{auth_db} <<\\EOF\ndb=db.getSiblingDB(\"{db_name}\");{slave_ok}printjson({sql})\nEOF".format(
mongo=mongo, uname=self.user, password=self.password, host=self.host, port=self.port, db_name=db_name, sql=sql, auth_db=auth_db, slave_ok=slave_ok)
cmd = "{mongo} --quiet -u {uname} -p '{password}' {host}:{port}/{auth_db} <<\\EOF\ndb=db.getSiblingDB(\"{db_name}\");{slave_ok}load('{tempfile}')\nEOF".format(
mongo=mongo, uname=self.user, password=self.password, host=self.host, port=self.port,
db_name=db_name, sql=sql, auth_db=auth_db, slave_ok=slave_ok, tempfile=tempfile)
else:
cmd = "{mongo} --quiet -u {user} -p '{password}' {host}:{port}/{auth_db} <<\\EOF\nrs.slaveOk();{sql}\nEOF".format(
mongo=mongo, user=self.user, password=self.password, host=self.host, port=self.port, db_name=db_name, sql=sql, auth_db=auth_db)
Expand All @@ -283,9 +292,11 @@ def exec_cmd(self, sql, db_name=None, slave_ok=''):
_re_msg.append(_line)

msg = '\n'.join(_re_msg)

except Exception as e:
logger.warning(f"mongo语句执行报错，语句：{sql}，{e}错误信息{traceback.format_exc()}")
# 删除mongo脚本临时文件
if os.path.exists(f'{tempfile}'):
os.remove(tempfile)
return msg

def get_master(self):
Expand Down