Releases: WeBankFinTech/DataSphereStudio
1.2.1
New Feature
-
Workflow data development nodes support referencing resource parameter templates.
-
The queue manager now includes Application information display.
-
Scriptis supports moving file directories.
-
Real-time workflow execution supports proxy users.
-
Workflow node descriptions will be displayed in the remarks column after being published to Schedulis.
-
Added code association condition prompt check.
-
The DAG mode of workflows adds a flowchart type.
-
Data services support the execution of multiple SQL queries and the download of multiple result sets.
-
Data services support viewing historical execution results.
-
Data services support Hive SQL query statements.
-
DSS supports configuring Starrocks data sources.
-
DSS workflow projects allow integration with Git management.
Enhancement
- Added a front-end page for announcement configuration.
- The table field information page in database details supports searching by field name, description, and other information.
- Allows some accounts to configure the full result set download feature.
- Python language server provides syntax localization in Chinese.
- Optimized the display of Spark script explain results.
- Result sets support column filtering and field width adjustment.
- Hive table field information supports copying all content.
- Scriptis file navigation bar supports dragging with a scrollbar.
- Workflow development page supports quick project creation.
- User auto-join workspace feature allows configuration by department.
- Cross-cluster queues need to be displayed in the resource manager.
- Added parameter configuration to signal receiving nodes, allowing signals to be accepted only if sent by signal nodes executed on the same day.
- Configurable automatic release rules for idle engines in cross-cluster queues.
- Hive workflow nodes add parameter configuration items for
mapreduce.job.running.map.limit
andmapreduce.job.running.reduce.limit
. - Optimized front-end display when there are too many result set columns.
- Project list name display optimization.
- Uniformity maintained for delimiter types in result set export scenarios.
- Workflow nodes now have a property to disable nodes.
- Trino script result sets support full download as Excel files.
- Datachecker nodes provide error prompts for view types.
- Optimized job.desc validation for Datachecker nodes.
- Improved query functionality on the database details page.
- Adjusted engine release time logic in resource templates.
- Resource templates support distribution by department.
- Optimized engine reuse logic to avoid using small resource engines for tasks requiring large resources.
- Optimized default values for whether to reference resource parameter templates.
- Added an administrator role for the table owner transfer function to handle ownerless data across various business department databases.
- Datachecker node
source.type
configuration allows clearing after setup. - Data service parameter lengths support user customization.
- Database details page now includes heat information based on recent table access times.
- Enhanced contrast for selected scripts.
- Cached front-end result set view states.
Bugs Fix
- Fixed the issue where the task progress bar did not match the task status.
- Fixed the issue where the date variable in the email subject of the sendemail node did not work during historical reruns.
- Fixed the issue where appconn failed to auto-refresh.
- Fixed the issue where the resource template was not applied when running scripts within workflow nodes.
1.1.2
DataSphereStudio1.1.2 has undergone significant optimization at the system level, such as merging and multi activity transformation of microservices. Simultaneously providing some new features.
DSS-1.1.2 includes all of Project DSS-1.1.2.
The main features of DSS-1.1.2 are as flollows:
- Merge microservices. Merging six microservices into two services, DSS-Server and DSS-Apps Server, greatly reducing the difficulty of deployment and troubleshooting problems.
- Support multi active deployment of microservices DSS Server and DSS Apps Server to achieve high system availability.
- Integration has been completed with the latest version 1.4.0 of Apache Linkis, allowing users to install and deploy two components with just one click.
- Supports elegant upgrades. Provided upgrade documentation on how to upgrade from DSS 1.1.1 to DSS 1.1.2.
- User experience optimization. Support for full screen display of result sets and logs, clearer exception information, and project information display after user login.
- Add workflow copying and import/export functions to facilitate user migration of workflows.
- Support users to manually release the editing lock of the workflow
Abbreviations:
- DSS: DataSphereStudio
New Feature
- DSS-1096 [DSS-Structure] Merge microservices. Merging six microservices into two services, DSS-Server and DSS-Apps Server, greatly reducing the difficulty of deployment and troubleshooting problems.
- DSS-1095 [DSS-Structure] Support multi active deployment of microservices DSS Server and DSS Apps Server to achieve high system availability.
- DSS-1103 [DSS-Commons] Complete integration with the latest version 1.4.0 of Apache Linkis.
- DSS-1104 [DSS-Deployment] Supports elegant upgrades from DSS1.1.1 to 1.1.2.
- DSS-1069 DSS-1065 [DSS-Workflow] Add workflow replication and import/export functions.
- DSS-1068 [DSS-Workflow] Support users to manually release the editing lock of the workflow.
- DSS-1060 DSS-1067 DSS-1073 [DSS-UI] Support full screen display of result sets and logs, with clearer front-end exception information, etc.
Enhancement
- DSS-1089 [DSS-Workspace] New bulletin board function, supporting regular or periodic announcement to users.
- DSS-1050 [DSS-Workspace] After entering the workflow editing page, the left project bar can switch to display personal projects and all projects.
- DSS-1053 [DSS-Workspace] Verify the project name when creating a new project.
- DSS-1058 [DSS-Workspace] The default display for project presentation is the one I created.
- DSS-1063 [DSS-Workspace] Add users to join two workspaces by default.
- DSS-1093 [DSS-Workspace] Optimization of workspace administrator permissions.
- DSS-1054 [DSS-Scriptis] Adjusting the digital display information in the Scriptis floating window from the number of tasks to the number of engines.
- DSS-1070 [DSS-Scriptis] Scriptis engine management page with arrow in the bottom right corner, add the Select All Idle Engines button.
- DSS-1079 [DSS-Scriptis] Optimization of Table Creation Wizard Verification Rules.
- DSS-1049 [DSS-Scriptis] Browser version detection alarm.
- DSS-1056 [DSS-Scriptis] Add TSQL (trino) type script.
- DSS-1051 [DSS-Commons] DSS log exception optimization, providing users with more comprehensive and accurate exception information.
- DSS-1052 [DSS-Commons] Third-party AppConn dependency optimization, managed by DSS uniformly for Apache Linkis packages on third-party dependencies.
- DSS-1057 DSS-1059 [DSS-Workflow] DSS workflow node DataChecker supports recognition of ${run_today_h}, ${yyyyMM}, and JobDesc supports lengths up to 5000.
- DSS-1097 DSS-1071 [DSS-Workflow] The task configuration of the workflow Spark node defaults to removing unit G, and supports scripts in the node to set Spark parameters.
- DSS-1062 [DSS-Workflow] Optimize transactions for importing and exporting orchestrator in the code.
- DSS-1080 [DSS-Workflow] In the workflow entry module, remove the default interceptor for Apache Linkis.
- DSS-1081 [DSS-AppConn] linkis-appconn-engine plugin module removes Apache Linkis dependencies.
- DSS-1085 [DSS-AppConn] Clean up downloaded lib packages when AppConn engine exits.
- DSS-1087 [DSS-AppConn] Add a new isMicroApp field to the AppConns interface to represent micro applications.
- DSS-1091 [DSS-Appconn] Linkis jobtype plugin log printing optimization.
- DSS-1055 [DSS-Commons] Regularly and automatically clean up ContextMaps and ContextIDs that are no longer used in the database.
- DSS-1064 [DSS-Commons] Add audit log printing for easy retention of user operation records.
- DSS-1077 [DSS-Commons] New user guidance optimization.
- DSS-1078 [DSS-Commons] When the version is released, the front-end prompts the user to refresh.
Bugs Fix
- DSS-1082 [DSS-Scriptis] Fix the issue of slow loading caused by repeated front-end calls to the same interface after the Scriptis page is refreshed.
- DSS-1083 [DSS-AppConn] Fix the issue of unclear error messages thrown by Schedulis.
- DSS-1072 [DSS-Appconn] Fix the issue with printing Schedulis logs, changing to only print the first 1024 characters.
- DSS-1090 [DSS-Workflow] Fix the issue of workflow and project names being too long.
- DSS-1086 [DSS-Workflow] Fix the issue of incomplete display of workflow node names.
- DSS-1061 [DSS-Workflow] Fix the issue where users cannot view the horizontal drag bar below the workflow editing interface.
- DSS-1049 [DSS-Web] Fix browser stuttering during use.
Credits
The release of DSS 1.1.2 is inseparable from the contributors of the DSS community. Thanks to their active participation in the community, DSS 1.1.2 was released smoothly.
Compiled Version
DSS1.1.2&Linkis1.4.0 Compiled package
Assets
1.1.1
DataSphereStudio 1.1.1 is an update to version 1.1.0, which mainly involves support for creating workflows without scheduling system integration, fixing the ability to publish scripts as RESTFUL APIs, fixing Schedulis, Qualitis, and DolphinScheduler password-free jump exceptions, fixing Qualitis node publishing to DolphinScheduler exceptions, fixing workspace management user list interface exceptions, and Fix the exception of workflow editing for publishing to DolphinScheduler.
But you have to note that the 'subFlow' node in DataSphereStudio cannot be published to DolphinScheduler at present.
DSS-1.1.1 includes all of Project DSS-1.1.1.
The main features of DSS-1.1.1 are as flollows:
- Support to create workflows without integrated scheduling system.
Abbreviations:
- DSS: DataSphereStudio
Enhancement
- DSS-993 [DSS-Workflow] Support to create workflow without integrated scheduling system.
Bugs Fix
- DSS-954 [DSS-Workspace] Fix the password-free jump exception of Schedulis and Qualitis.
- DSS-984 [DSS-Workspace] Fix the password -free jump exception of DolphinScheduler.
- DSS-967 [DSS-Workspace] Fix the problem of abnormality in the interface of getting workspace management user list.
- DSS-991 [DSS-Appconn] Fix the exception of publishing the Quality node to DolphinScheduler.
- DSS-997 [DSS-Appconn] Fix workflow editing exception when it has been published to DolphinScheduler.
- DSS-995 [DSS-Scriptis] Fix the ability to publish scripts as RESTFUL API.
Credits
The release of DSS 1.1.1 is inseparable from the contributors of the DSS community. Thanks to all the community contributors, Including but not limited to the following contributors:ichenfeiyang、jacktao007、LMQ0023、utopianet、coombe。
Compiled Version
Tencent Cloud:
DSS-1.1.1&Linkis-1.1.1 Compiled package (.tar.gz)
DSS-1.1.1 Compiled package (.tar.gz)
1.1.0
DataSphereStudio 1.1.0 is a milestone in the practice of data application development and management framework. It integrates all the open source ecological components of WeDataSphere, and brings a series of powerful new features, as well as a more streamlined and easy-to-connect data application development integration architecture design and implementation .
DSS-1.1.0 includes all of Project DSS-1.1.0.
The main features of DSS-1.1.0 are as follows:
- All the open source ecological components of WeDataSphere have been integrated, including Apache Linkis 1.1.1, Exchangeis 1.0.0, Schedulis 0.7.0, Qualitis 0.9.2, Visualis 1.0.0, Streamis 0.2.0 and Prophecis 0.3.2.
- Integrate Apache DolphinScheduler 1.3.X. It supports one-click publishing of DSS workflows as DolphinScheduler workflows, designs and develops a new scheduling center for workflow scheduling.
- User experience optimization. Such as support for skinning, top navigation bar revision, DSS development center revision, etc.
- Help manual and beginner's guide.
- Installation and deployment optimization. It further simplifies the one-click installation and deployment process of DSS&Linkis family buckets, allowing the installation of DSS and Linkis to be completed within half an hour.
- Support graceful upgrade. Provides a detailed upgrade process on how to upgrade from DSS-1.0.1 to DSS-1.1.0, and how to migrate from DSS-0.9 to DSS-1.1.0.
- AppConn architecture optimization. The architecture is simpler and clearer, and the documentation is more comprehensive and detailed. It teaches you how to implement a new AppConn and add a new workflow node.
Abbreviations:
- DSS: DataSphereStudio
- DAS: DataApiService
New Feature
- DSS-848 [DSS-AppConn] Integrate Apache DolphinScheduler1.3.X, design and develop a new scheduling center
- DSS-849 [DSS-Commons] Adapt to Apache Linkis1.1.1
- DSS-716 [DSS-AppConn] Integrate Exchangeis1.0.0
- DSS-846 [DSS-AppConn] Integrate Schedulis0.7.0
- DSS-847 [DSS-AppConn] Integrate Qualitis0.9.2
- DSS-850 [DSS-AppConn] Integrate Visualis1.0.0
- DSS-851 [DSS-AppConn] Integrate Streamis0.2.0
- DSS-852 [DSS-AppConn] Integrate Prophecis0.3.2
- DSS-853 [DSS-AppConn] AppConn Architecture Optimization
- DSS-792 [DSS-Deployment] Support hot update AppConn plugin installation without restarting all services
- DSS-862 [DSS-Deployment] Support graceful upgrade
- DSS-863 [DSS-Workspace] Added help manual and beginner's guide
- DSS-590 [DSS-UI] Optimize development center user experience, such as support changing the background and optimizing the display of the top navigation bar, etc.
- DSS-864 [DSS-Workflow] Add the script file download function of the workflow version
Enhancement
- DSS-865[DSS-Workspace] Streamline part of the interface of the workspace, while removing the useless part of the interface
- DSS-866 [DSS-Workspace] Add a workspace type option when creating a new workspace; add a management console function on the right side of the workspace homepage; add a help button in the lower right corner
- DSS-753 [DSS-Workspace] Remove the application store and change the component access entry to the upper left menu bar
- DSS-728 [DSS-Workspace] Optimize the demo of the application development process at the bottom of the homepage, remove the button of the demo case and change it to stay tuned
- DSS-590 [DSS-UI] Adjust the style of the explorer button, add the background switching function, change the UI of the project list and navigation bar, change the UI style of the application store
- DSS-628 [DSS-DAS] The download function is limited to 5000, and every download operation will pop up an operation risk prompt to the user
- DSS-780 [DSS-DAS] Optimiz result set visualization page display, optimize result set paging sorting, support result set and log part drop-down, support result set table width extension
- DSS-864 [DSS-Workflow] Add workflow basic property display
- DSS-724 [DSS-Workflow] Automatically check the workflow mode and workflow mode when creating a new workflow
- DSS-725 [DSS-Workflow] Add the workflow importance level feature when creating a new workflow
- DSS-161 [DSS-Workflow] Support the use of runtime variables in datachecker
- DSS-868 [DSS-Scriptis] Support dragging the code tab to any position
- DSS-869 [DSS-Scriptis] Support mutual modification of Sql and Hql file types
- DSS-870 [DSS-Scriptis] Support copy and paste workspace code files
- DSS-720 [DSS-Scriptis] Support only viewing tables created by yourself
- DSS-871 [DSS-Commons] Help users clear cache after login
- DSS-872 [DSS-AppConn] Supplement Update and Delete operations of Schedules AppConn, and add corresponding interfaces
Bugs Fix
- DSS-650 [DSS-Workspace] Fixed an issue where all 3rd party system items could not be deleted synchronously when deleting an item
- DSS-594 [DSS-Workspace] Fix the problem of 400 bad request reported in workspace management-user management, editing user interface
- DSS-873 [DSS-Workspace] Fixed the problem that the user data displayed abnormally on the user management page
- DSS-601 [DSS-Workspace] Fixed the issue that the drop-down boxes of publish permission, edit permission and view permission on the Create Project page could not obtain all users of the workspace
- DSS-632 [DSS-Workspace] Fixed the problem that the file directory displayed in the front-end workspace was incorrect after switching the proxy user
- DSS-874 [DSS-Workspace] Fixed the problem that the interface did not display the visual interface after clicking the visual button
- DSS-621 [DSS-Workspace] Fix the problem that the script selects right-click to open to the side console and reports an error
- DSS-875 [DSS-Workspace] Fix the problem of copying and pasting the script to the first opened folder and reporting an error
- DSS-744 [DSS-Workspace] Fix the problem that other users have abnormal permissions after a user grants permissions to other users after creating a new project
- DSS-704 [DSS-Workspace] Fix the problem that the project copy function button is not displayed
- DSS-667 [DSS-Workspace] Fix the problem that the request to the third-party application to check whether the project name is duplicated fails when creating a project
- DSS-652 [DSS-Workflow] Fix the problem that the eventsender and eventreceiver nodes fail to pass parameters
- DSS-640 [DSS-Workflow] Fixed the issue that new built-in parameters such as run_year in datachecker did not take effect
- DSS-759 [DSS-Workflow] Fix the problem that the exported data is disordered or lost when there is a subflow
- DSS-695 [DSS-Workflow] Fix the issue that sendmail node fails to run
- DSS-692 [DSS-Workflow] Fix the failure of workflow rollback in the development center
- DSS-600 [DSS-Workflow] Fix the Fai...
1.0.1
DSS-1.0.1 includes all of Project DSS-1.0.1.
DSS-1.0.1 mainly contains three improvements and enhancements:
- Adapt to Apache Linkis 1.0.3.
- Deprecate Jersey and use Spring MVC to build HTTP RESTful APIs.
- Optimize log printing of one click installation deployment script.
Abbreviations:
DSS: DataSphereStudio
New Feature
- DSS-444 [DSS-Eco] Adapt to Apache Linkis 1.0.3.
- DSS-445 [DSS-Architecture] Deprecate Jersey and use Spring MVC to build HTTP RESTful APIs.
Enhancement
- DSS-448 [DSS-Package] Optimize log printing of one click installation deployment script.
- DSS-521 [DSS-Document] Update README about the compatibility information of DSS with integrated third-party application tools.
Bugs Fix
- DSS-499 [DSS-Execution] Modify the reuse configuration parameters of the appconn engine to solve the problem that the appconn engine cannot be reused.
- DSS-516 [DSS-Execution] Add the kill interface to solve the problem of cannot successfully kill a workflow.
- DSS-470 [DSS-Apiservice] Modify the log printing class referenced by apiservcice module to solve the problem of module compilation error.
- DSS-448 [DSS-Package] Update the DDL of the dss application component to solve the problem of incorrect display of component list data.
- DSS-448 [DSS-AppConn] Modify the parameters of the AppConnEngineConnExecutor interface to solve the exception that occurs when the workflow node is deleted.
- DSS-448 [DSS-Standard] Optimize the judgment logic of SSO reuse login status to solve the problem that the login cannot be redirected normally.
- DSS-476 [DSS-Workflow] Modify the format of the original interface request parameters to solve the problem of error reporting in workspace creation.
- DSS-521 [DSS-Orchestrator] Resolve the compilation problem about Pair class.
Credits
The release of DSS 1.0.1 is inseparable from the contributors of the DSS community. Thanks to all the community contributors!
Compiled Version
- Tencent Cloud:
1.0.0
DSS-1.0.0 includes all of Project DSS-1.0.0.
DataSphereStudio 1.0.0 is a major release marking the start of data application development and management framework, which brings both a variety of powerful new features and strong archetecture designation and implemenation of data application.
The following key features are added:
- Add dss-standard module, which is three kind of basic request protocol to integrate with upper-layer application systems.
- Provides the ability to obtain and manage orchestrator instances.
- Add workflow conversion standard module, provide the capability of converting a DSS workflow to the tasks of third-party application tools, such as the workflow of Schedulis.
- Define AppConn core interface and default implementation class.
- Provide the CRUD interface of the project in the framework module.
- Provide the CRUD interface of the workspace in the framework module.
- Supports data API service
- Enhance the capability of DSS workflow execution server, adapt to the new architecture of DSS1.0.0
- Optimize the packaged deployment module of DSS1.0.0
- Compatible with linkis-1.0.2 and above
- Add a new front-end interface of DSS1.0.0
Abbreviations:
DSS: DataSphereStudio
New Feature
Framework
- DSS-364 [DSS-Framework] Provide common classes of framework modules.
- DSS-365 [DSS-Framework] Provide CRUD interface for orchestration mode.
- DSS-366 [DSS-Framework] Provide the CRUD interface of the project in the framework module.
- DSS-367 [DSS-Framework] Provide the CRUD interface of the workspace in the framework module.
- DSS-368 [DSS-Framework] Enhance the ability of the framework and provide the function of orchestration mode release.
- DSS-369 [DSS-Framework] Define some public classes of the orchestration module.
Orchestrator
- DSS-352 [DSS-Orchestrator] Define some common classes for the orchestration module.
- DSS-353 [DSS-Orchestrator] Define relevant interface specifications for orchestrator mode conversion.
- DSS-354 [DSS-Orchestrator] Defines the basic class of the orchestration module.
- DSS-355 [DSS-Orchestrator] Provides the orchestrator module with the ability to access database persistence.
- DSS-356 [DSS-Orchestrator] Provides the ability to obtain and manage orchestrator instances.
Workflow
- DSS-349 [DSS-Workflow] Provide DSS workflow commons module.
- DSS-350 [DSS-Workflow] Add workflow conversion standard module.
- DSS-361 [DSS-Workflow] Provide DSS workflow sdk module
- DSS-362 [DSS-Workflow] Enhance the capability of DSS workflow server,adapt to the new architecture of DSS1.0.0
Standard
- DSS-370 [DSS-Standard] Defines the third-level development standard of AppConn
- DSS-371 [DSS-Standard] Provide DSS standard commons module
- DSS-372 [DSS-Standard] Defines the second-level development standard of AppConn
- DSS-373 [DSS-Standard] Defines the first-level development standard of AppConn
- DSS-384 [DSS-Standard] Supports the integration of third-party applications using DSS standard
AppConn
- DSS-351 [DSS-AppConn] Realize the application of the three major access specifications of DSS1.0.0
- DSS-357 [DSS-AppConn] Define AppConn core interface and default implementation class.
- DSS-358 [DSS-AppConn] Implement DSS1.0.0 AppConn class loading and instantiation module.
- DSS-359 [DSS-AppConn] Define the abstract dispatch AppConn.
- DSS-360 [DSS-AppConn] Implement Linkis AppConn plugin.
- DSS-363 [DSS-AppConn] Provide the ability of AppConn and AppInstance to persist the database.
- DSS-377 [DSS-AppConn] Provide some basic function modules of appconn in DSS1.0.0
ApiService
- DSS-395 [DSS-ApiService] Supports data API service
Web
- DSS-376 [DSS-Web] Add a new front-end interface of DSS1.0.0
Enhancement
- DSS-347 [DSS-Workflow] Enhance the capability of DSS workflow execution server.
- DSS-348 [DSS-Workflow] Enhance the capability of DSS workflow node execution module.
- DSS-362 [DSS-Workflow] Enhance the capability of DSS workflow server, adapt to the new architecture of DSS1.0.0
- DSS-380 [DSS-DataPipe] The data import and export service module provides an interface for data import and export.
- DSS-381 [DSS-DataPipe] Provides the module for data import and export
- DSS-388 [DSS-Package] Optimize the packaged deployment module of DSS1.0.0
- DSS-390 [DSS-Config] Update the configuration file and database script file
Bugs Fix
- DSS-385 [DSS-Package] Modify the packaged deployment module of DSS1.0.0
- DSS-389 [DSS-Package] Update the configuration file and database script file of DSS1.0.0
- DSS-393 [DSS-ContextService] Add an interface to get ContextID
- DSS-396 [DSS-Package] Remove some useless code and solve compilation problems
- DSS-399 [DSS-Package] Optimize application startup script
Credits
The release of DSS 1.0.0 is inseparable from the contributors of the DSS community. Thanks to all the community contributors!
Compiled Version
- Tencent Cloud:
0.9.1
DSS 0.9.1 is a version led by ChinaTelecom Ctyun Big Data Platform Team with the help of WeBank.
This is the next release based on DSS 0.9.0 . This version aims to reduce the operation and maintenance costs of WeDataSphere components such as DSS, Linkis, and Schedulis for community users through the new feature of "new user initialization", so that DSS can use "new user initialization" to automatically initialize the execution environment of new created users.
Since 0.9.0, this version contains 7 improvements and enhancements.
Enhancement
-
[DataSphereStudio-274] Adding account creation function. Support users to customize configuration according to their own environment,currently available to create include(LDAP,management node account,HDFS&Linux directory ,scheduling account,hive database,keytab), because the environments of community users are quite different, users can also add new implementations or modify scripts to adapt to their own environment.
-
[DataSphereStudio - 288] Support creating users on the master machine
-
[DataSphereStudio - 289] Support creating workspace for the newly added user(support Linux and HDFS) and HDFS
-
[DataSphereStudio - 290] Support adding users to the Schedulis configuration and immediately overriding.
-
[DataSphereStudio - 291] Support initializing the environment needed by Hive for newly added users.
-
[DataSphereStudio - 292] Support for creating and distributing keytab files in Kerberos clusters for new users.
-
[DataSphereStudio - 293] Support adding users to the LDAP based system
Credits
The release of DSS 0.9.1 is inseparable from the contributors from the WeDataSphere community. They selflessly contribute their code and actively communicate with community partners. Without their help, DSS 0.9.1 cannot be successfully released. Thanks to all the community contributors! Rank in no particular order:
luxl : Basic code and workspace module, scheduling authorization, and part of the front-end development of account creation.
HanTang:LDAP module and Hadoop master node account creation function.
lvjw: Documentation contribution.
schumiyi : Fix the problem that the application prompts that it is not open source after the initial installation.
tomshy1 : Front-end development of account creation .
JsonLiuUp: Keytab creation and distribution, hive database activation and authorization.
ldtong:Scheduling authorization.
Upgrade Guide
This version involves adjustments to some JARs. If DSS is running in your production environment, you only need to simply replace some of DSS Jars to upgrade. For details, please refer to: DSS 0.9 .1 Upgrade Guide.
Please note: This version provides a new feature called "new user initialization" . In order to use this feature properly, please keep notice of DSS0.9.1 New User Initialization Usage Document.
Cloud Resource
DSS + Linkis + Qualitis + Visualis + Azkaban one-click installation package
URL:dss_linkis.zip
DSS 0.9.1 是在微众银行的倾力帮助下,由 天翼云 主导完成的一个版本。
这是基于 DSS 0.9.0 的下一个发行版本。该版本旨在通过新增“新用户初始化”特性,为社区用户降低运维DSS、Linkis和Schedulis等WeDataSphere组件的运维成本,让DSS在新增新登录用户时,可以使用“新用户初始化”功能自动完成新用户的所有环境初始化操作。
本次版本包含了7个改进和增强。
特性增强
- [DataSphereStudio-274] 新增新用户初始化特性。不同的环境,支持配置化创建新用户所需的所有环境信息,目前包括:LDAP、管理节点账号、HDFS和Linux目录、调度账号、hive库、keytab等,且预留了相关接口,允许用户自己新增实现,或者修改脚本适配自己的环境。
- [DataSphereStudio-288] 新增在主控机上创建用户特性。
- [DataSphereStudio-289] 支持给新用户创建工作空间(支持Linux和HDFS)。
- [DataSphereStudio-290] 支持在Schedulis的配置中添加用户,并立刻重载。
- [DataSphereStudio-291] 支持给新用户初始化HIVE所需环境信息。
- [DataSphereStudio-292] 支持为新用户在Kerberos集群创建和分发keytab文件。
- [DataSphereStudio-293] 支持给基于LDAP体系新增用户。
新贡献者
DSS 0.9.1 的发布,离不开WeDataSphere社区的贡献者,他们无私地贡献自己的代码,积极地与社区伙伴进行技术交流,有了他们的助力,DSS 0.9.1 才能顺利地发布,在此感谢各位社区的贡献者! 排名不分先后:
luxl:本次版本的社区主导者。基础代码和工作空间模块,调度授权,以及创建账号部分前端开发。
HanTang:LDAP模块和hadoop主节点账号创建功能。
lvjw:贡献文档。
schumiyi:修复初始化安装后进入应用提示未开源的问题。
tomshy1:创建账号前端开发。
JsonLiuUp:keytab创建和分发,hive库开通和授权。
ldtong:调度授权。
升级指南
本次版本涉及对一部分JAR包进行了调整,如果您已经在生产环境使用了DSS,并且不想重新安装DSS的话,只需要对其中的几个Jar进行简单替换即可,具体请参考:DSS 0.9.1升级指南。
请注意:本次版本提供了新用户初始化的大特性,为了使您能正常使用该特性,请在安装和升级前,先阅读 DSS0.9.1新用户初始化使用文档。
云资源
DSS + Linkis + Qualitis + Visualis + Azkaban一键安装包
链接:dss_linkis.zip
0.9.0
0.9.0 is an important version led by ChinaTelecom Ctyun Big Data Platform Team with the help of WeBank.
This is the next release based on DataSphere Studio 0.8.0 line. This version aims to build a one-stop big data application development and management platform by introducing the concepts of "DSS Integration Standard" and "Workspace", which greatly enhances the user's big data development experience.
DSS Integration Standard (DSS Integration Standard): It is the unified access specification of the application system composed of SecurityService from the AppJoint specification of DSS, so that enterprises can connect related big data products to DSS very easily One-stop big data application system portal display and management capabilities (DSS application access specifications will continue to be supplemented and improved in subsequent versions of DSS).
Workspace specification: enables different types of data applications (such as workflow offline applications, real-time applications, data API services, etc.) to be organized and managed from a unified perspective, enhancing application organization and management capabilities.
It contains 23 bug fixes, improvements and enhancements.
Enhancement
- [DSS-149] DSS Homepage optimization, mainly includes management items for operation and workspace creation, as well as cases and introductions.
- [DSS-150] DSS Workspace Homepage optimization. After the administrator creates a space for the corresponding role (for example, developer), this page is the first page after login. Includes entries for common applications, administrator functions, and various types of applications.
- [DSS-151] Common function configuration optimization of the workspace. The roles provided in the workspace can be freely configured and add common function entries that can be added and deleted.
- [DSS-152] Optimize DSS front-end joint adjustment configuration. After adding this configuration, you can modify the back-end gateway addresses in different environments of the agent as needed, such as production and testing, to quickly locate the problem.
Bug Fix
- [DSS-120] add use cases such as HQL, PySpark, Spark SQL for new users, so that new users can test and use.
- [DSS-176] footer-channel in iframe not work.
- [DSS-185] 0.8.0-scriptis cannot show the execution result of python scripts.
- [DSS-193] Replace DWS with DSS.
Credits
The release of DataSphere Studio 0.9.0 is inseparable from the contributors of the WeDataSphere community. They selflessly contribute their codes and actively carry out technical exchanges with community partners. With their help, DataSphere Studio 0.9.0 can be successfully released. Thank you all Contributors to the community! Rank in no particular order:
det101: contributed front-end codes for workspace homepage, optimized local debugging configuration of front-end, and solved the bug that footer-channel in iframe didn't work.
yuchenyao: contributed front-end codes for DSS homepage, and optimized the process of creating project and workflow.
schumiyi: contributed back-end codes for DSS homepage and workspace homepage.
ryanqin01: contributed database related codes。
AdamWang: contributed back-end codes for workspace homepage, and reconstructed part of remaining problems.
Upgrade Wizard
Due to the addition of the concept of user workspace (Workspace) in this version of DSS-0.9.0, if you are upgrading from DSS0.7 or DSS0.8 to DSS0.9.0, after completing the platform deployment, you need to make some adjustments to the database tables , Please visit: DSS 0.9.0 Upgrade Guide
Cloud Resource
- DSS one-click installation package
URL: https://pan.baidu.com/s/1unf0FqO6GrvUzYjnHOBWug Password:nnx9 - DSS + Linkis + Qualitis + Visualis + Azkaban one-click installation package
URL:https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/DataSphereStudio/0.9.0/dss_linkis.zip - dss-web one-click installation package
URL: https://pan.baidu.com/s/1nS4iVildF4oXFdd1CPh3AA Password: 56nh - linkis-jobType(Azkaban)
URL:https://pan.baidu.com/s/1hkCaVFoIWJH5atedSqhd8Q Password:09pz
0.9.0是在微众银行的倾力帮助下,由天翼云大数据平台团队主导完成的一个重要版本。
这是基于DataSphere Studio 0.8.0的下一个发行版本。该版本旨在通过引入“DSS应用接入规范”,“工作空间”等概念,构建一站式大数据应用开发与管理平台,极大提升了用户的大数据开发体验。
DSS应用接入规范(DSS Integration Standard): 是从DSS的AppJoint规范中,抽出SecurityService组成的应用系统统一接入规范,使企业可以非常简单地将相关大数据产品接入到DSS之中,快速具备一站式的大数据应用系统门户展示和管理能力(DSS应用接入规范将会在DSS的后续版本之中继续补充完善)。
工作空间规范:使不同类型的数据应用(如:工作流离线应用、实时应用、数据API服务等)能够以统一的视角进行组织管理,提升应用的组织和管理能力。
本次版本包含了23个Bug修复,改进和增强。
特性增强
- [DSS-149] DSS首页优化,主要包括运营入口和工作空间创建管理入口,也包括案例和入门。
- [DSS-150] DSS工作空间首页优化,管理员创建空间分配给对应的角色之后,如开发者,那么该页面就是登陆之后的首页。
- [DSS-151] 工作空间常用功能配置,提供给工作空间的角色自由配置增加常用的功能入口,可以增加和删除
- [DSS-152] dss前端联调配置优化,增加该配置之后,可以根据需要修改代理不同环境的后端gateway地址,如生产和测试,快速定位问题。
Bug修复
- [DSS-120] 为新用户使用增加包括HQL、PySpark、Spark SQL等用户用例,方便新用户测试试用。
- [DSS-176] 列表浮动块在iframe下面拖动存在问题。
- [DSS-185] 0.8.0-scriptis执行python脚本无法显示结果。
- [DSS-193] 历史遗留问题重构,将DWS替换为DSS。
新贡献者
DataSphere Studio 0.9.0的发布,离不开WeDataSphere社区的贡献者,他们无私地贡献自己的代码,积极地与社区伙伴进行技术交流,有了他们的助力,DataSphere Studio 0.9.0才能顺利地发布,在此感谢各位社区的贡献者! 排名不分先后:
luxl :贡献工作空间功能的前端代码,优化前端本地调试配置,修复任务管理器入口在iframe下拖动不正常的问题
yuchenyao:贡献了首页的前端代码,优化了创建工程及工作流的流程。
schumiyi: 贡献了首页和用户工作空间相关后端代码。
ryanqin01: 贡献了数据库相关代码。
AdamWang:贡献了用户工作空间相关后端代码,以及部分历史代码遗留问题重构。
升级向导
由于本次DSS-0.9.0版本新增了用户工作空间(workspace)概念,如果您是从 DSS0.7 或 DSS0.8 升级到 DSS0.9.0,在完成平台部署后,需对数据库表做一些调整,具体请访问: DSS 0.9.0 升级指南
云资源
- dss一键安装包
链接:https://pan.baidu.com/s/1unf0FqO6GrvUzYjnHOBWug 提取码:nnx9 - DSS + Linkis + Qualitis + Visualis + Azkaban一键安装包
链接:https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/DataSphereStudio/0.9.0/dss_linkis.zip - dss 前端安装包
链接: https://pan.baidu.com/s/1nS4iVildF4oXFdd1CPh3AA 提取码: 56nh - linkis-JobType(Azkaban)
链接:https://pan.baidu.com/s/1hkCaVFoIWJH5atedSqhd8Q 提取码:09pz
0.8.0
Enhancement
- [DSS-158] Database - > Table - > query table. The pop-up temporary file name is dbName.tableName.hql Change to dbname_ tableName.hql.
- [DSS-168] Update DataSphereStudio with linkis version 0.9.4 and solves some compilation errors
Bug Fix
- [DSS-107] Fixed the bug that multiple ips are gotten.
- [DSS-116] Fixed the eventchecker plugin in the appjoint/scheduler module will have repeated signal consumption in extreme scenarios.
- [DSS-157] Fixed the result set sorting error. Only the current column is sorted in ascending or descending order.
- [DSS-159] Fixed eventcheck node parameters msg.body default setting error.
- [DSS-160] Fixed datacheck cannot check the table correctly when the partition format is incorrectly filled in.
- [DSS-161] Fixed run_date variable can not be shown correctly.
- [DSS-162] Fixed flow runtime execute cost time is not correct.
- [DSS-165] Fixed the bug that the script in the workflow has been executed, but the execution time has been increasing.
- [DSS-167] Fixed 'utf-8' is not a valid value for the Content-Encoding header.
Credits
The release of DataSphereStudio0.9.4 is inseparable from the contributors of the WeDataSphere community. They selflessly contribute their codes and actively carry out technical exchanges with community partners. With their help, DataSphereStudio0.9.4 can be successfully released. Thank you all Contributors to the community! Rank in no particular order:
5herhom: Fixed bug with multiple IP addresses on the server that would cause installation and startup to fail.
xing.huang: Contributed error using 'UTF-8' as content of the Content-Encoding.
SelfImpr001:Contribute to the optimized message consumption code of eventcheck node.
特性增强
- [DSS-158] 数据库->表->查询表,弹出的临时文件名为dbName.tableName.hql改为dbName_tableName.hql
- [DSS-168] 更新DataSphereStudio的Linkis版本为0.9.4,并解决一些兼容问题。
Bug修复
- [DSS-107] 修复安装脚本中遇到环境存在多个IP时出错问题。
- [DSS-116] 修复EventCheck在特殊场景下重复消费消息问题。
- [DSS-157] 修复结果集排序错误问题,只对当前列进行了升序或者降序。
- [DSS-159] 修复EventCheck节点参数msg.body默认值设置错误问题。
- [DSS-160] 修复DataCheck节点因DataObject参数格式错误检查出错问题。
- [DSS-161] 修复DataCheck节点的run-date变量在前台日志中没有正确显示问题。
- [DSS-162] 修复工作流实时执行节点显示执行时间不正确。
- [DSS-165] 修复工作流脚本已经执行完成,但执行时间一直在增加问题。
- [DSS-167] 修复使用错误的'utf-8'作为Content-Encoding内容的问题。
新贡献者
DataSphereStudio 0.8.0的发布,离不开WeDataSphere社区的贡献者,他们无私地贡献自己的代码,积极地与社区伙伴进行技术交流,有了他们的助力,DataSphereStudio 0.8.0才能顺利地发布,在此感谢各位社区的贡献者! 排名不分先后:
5herhom: 修复了服务器存在多个IP会导致安装启动失败的BUG。
xing.huang: 贡献了错误使用'utf-8'作为Content-Encoding内容的修复代码。
SelfImpr001: 贡献了EventCheck优化消息消费方式的代码。
云资源
我们提供了DSS + Linkis + Qualitis + Visualis + Azkaban【全家桶一键部署安装包】,由于安装包过大(1.7GB),Github下载缓慢,请通过以下方式获取:
1. Baidu cloud:
URL: https://pan.baidu.com/s/1Jhq1vQB_gkYfbU-LMaUJJA
Password: 3hvj
以下为DSS后端安装包资源:
2. Baidu cloud:
URL: https://pan.baidu.com/s/1cbmd4rcjCYh4agZ26lsr2A
Password: 3l1r
以下为DSS前端安装包资源:
3. Tencent Cloud:
URL: https://share.weiyun.com/dJjSOAzE
Password: ur580z
DSS&LINKIS 一键部署脚本
Tencent Cloud:
0.7.0
Enhancement
- [DSS-25] Workflow supports python node type.
- [DSS-97] Workflow supports shell node type.
- [DSS-73][DSS-78] Workflow supports JDBC node type.
- [DSS-90] (Epic Enhancement) One-click deployment for DSS + Linkis + Qualitis + Visualis + Azkaban.
- [DSS-37] Add steps for environmental checks and service presence checks to DSS installation, and simplify the installation steps of the lite version.
- [DSS-79] Optimized the one-click deployment script and added common variables, so that DSS independent deployment and full one-click deployment are not affected.
- [DSS-88] The deployment user is added to the authentication configuration token.properties of dss-server by default.
- [DSS-94] DSS Web installation optimization.
Bug Fix
- [DSS-26] After the Sendemail node is deleted, a NoSuchElementException will be issued in releasing project.
- [DSS-56] When the node content is empty, it will cause the exception of deleting the visualis node to fail.
- [DSS-80] Fixed an exception that caused DSS to fail to start after Linkis-common removed the httpclient dependency.
- [DSS-81] Fix wrong JDBC node type definition in database.
- [DSS-82] Fixed an error that EventReceiver nodes shared custom variables, and an exception that the number of status polling threads would increase infinitely.
- [DSS-84] When visualis-server is started up in remote address, the status check will fail since the IP address is not converted to the remote actual IP address.
- [DSS-86] Fix using "hostname -i" to get local host IP error in centos8.
特性增强
- [DSS-25] 工作流新增 Python 节点。
- [DSS-97] 工作流新增 Shell 节点。
- [DSS-73][DSS-78] 工作流新增 JDBC 节点。
- [DSS-90] 【史诗级强化】DSS + Linkis + Qualitis + Visualis + Azkaban一键部署。
- [DSS-37] 安装DSS增加环境检查、服务存在性检查,并简化精简版安装步骤。
- [DSS-79] 优化一键部署脚本,新增通用变量,使DSS独立部署和全一键部署都受不影响。
- [DSS-88] 默认将部署用户添加至dss-server的认证配置token.properties中。
- [DSS-94] DSS Web安装优化。
Bug修复
- [DSS-26] 修复Sendemail节点删除后,发布工程会出现NoSuchElementException异常。
- [DSS-56] 修复节点内容为空时,会导致删除visualis节点失败的异常。
- [DSS-80] 修复Linkis-common移除httpclient依赖后会导致DSS无法启动的异常。
- [DSS-81] 修复数据库中JDBC节点类型定义错误。
- [DSS-82] 修复EventReceiver节点共享自定义变量错误,以及其状态轮询线程数会无限增加的异常。
- [DSS-84] 修复visualis-server远程启动后,因IP地址未转换成远程实际IP地址,会导致状态检测失败。
- [DSS-86] 修复centos8中使用”hostname -i” 获取本地主机IP错误。
云资源
我们提供了DSS + Linkis + Qualitis + Visualis + Azkaban【全家桶一键部署安装包】,由于安装包过大(1.3GB),Github下载缓慢,请通过以下方式获取:
1. Baidu cloud:
URL: https://pan.baidu.com/s/1hmxuJtyY72D5X_dZoQIE5g
Password: p82h
2. Tencent Cloud:
URL: https://share.weiyun.com/5vpLr9t
Password: upqgib
以下为DSS安装包资源:
1. Tencent Cloud:
URL: https://share.weiyun.com/5n2GD0h
Password: p8f4ug
2. Baidu cloud:
URL: https://pan.baidu.com/s/18H8P75Y-cSEsW-doVRyAJQ
Password: pnnj
DSS&LINKIS 一键部署脚本
Tencent Cloud: