Python SQL Parser and Transpiler

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Python interface to Hive and Presto. 🐝

More than 2000+ Data engineer interview questions.

Uses tokenized query returned by python-sqlparse and generates query metadata

Better SQL in Jupyter. 📊

Datafaker is a large-scale test data and flow test data generation tool. Datafaker fakes data and inserts to varied data sources. 测试数据生成工具

基于 scrapy-redis 的通用分布式爬虫框架

🎉🎉🐳 Datawhale大数据处理导论教程 | 大数据技术方向的开篇课程🎉🎉

大数据分析可视化实践

An Eyeriss Chip (researched by MIT, a CNN accelerator) simulator and New DNN framework "Hive"

Lightweight SQL DDL parser for extracting tables, columns, and schema metadata with broad multi-dialect support (HQL, TSQL, AWS Redshift, BigQuery, Snowflake and other dialects)

LazyOwn RedTeam/APT Framework is the first RedTeam Framework with an AI-powered C&C, featuring rootkits to conceal campaigns, undetectable malleable implants compatible with Windows/Linux/Mac OSX, and self-configuring backdoors. With its Web interface and powerful Console Client, it is the best combination for your Autonomous RedTeam/APT campaigns.

Example code for running Spark and Hive jobs on EMR Serverless.

This project aims to build an enterprise-grade offline data warehouse solution based on e-commerce platform order data.

python ETL framework

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

Educational notes,Hands on problems w/ solutions for hadoop ecosystem

a set of scripts to pull meta data and data profiling metrics from relational database systems

使用 python 操作大数据的各种组件