Skip to content

Commit

Permalink
Add bullmq for events handling
Browse files Browse the repository at this point in the history
Add docker config with redis
  • Loading branch information
PerminovEugene committed Nov 11, 2024
1 parent 797e553 commit 43e6220
Show file tree
Hide file tree
Showing 36 changed files with 754 additions and 259 deletions.
6 changes: 6 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,9 @@ DATABASE_NAME="replica"
PINATA_API_KEY=""
PINATA_API_SECRET=""
PINATA_JWT=""

REPLICA_SERVER_ADDRESS="http://localhost:3001"
ALLOWED_ORIGINS=""

REDIS_PORT=6379
REDIS_HOST=redis
Binary file modified .yarn/install-state.gz
Binary file not shown.
15 changes: 15 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,13 @@ services:
- NODE_ADDRESS=${NODE_ADDRESS}
- PINATA_JWT=${PINATA_JWT}
- IPFS_GATEWAY=${IPFS_GATEWAY}
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS}
- REDIS_PORT=${REDIS_PORT}
- REDIS_HOST=${REDIS_HOST}
depends_on:
- postgres
- ui
- redis
restart: on-failure
networks:
- hardhat-net
Expand All @@ -102,8 +106,19 @@ services:
- ./packages/replica-server:/app/packages/replica-server
- ./shared:/app/shared

redis:
image: redis:6-alpine
ports:
- "6379:6379"
command: redis-server --save 60 1 --loglevel warning
volumes:
- redis-data:/data
networks:
- hardhat-net

volumes:
postgres_data:
redis-data:

networks:
hardhat-net:
Expand Down
101 changes: 3 additions & 98 deletions packages/replica-server/README.md
Original file line number Diff line number Diff line change
@@ -1,99 +1,4 @@
<p align="center">
<a href="http://nestjs.com/" target="blank"><img src="https://nestjs.com/img/logo-small.svg" width="120" alt="Nest Logo" /></a>
</p>
# Contract data replication server

[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
[circleci-url]: https://circleci.com/gh/nestjs/nest

<p align="center">A progressive <a href="http://nodejs.org" target="_blank">Node.js</a> framework for building efficient and scalable server-side applications.</p>
<p align="center">
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/v/@nestjs/core.svg" alt="NPM Version" /></a>
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/l/@nestjs/core.svg" alt="Package License" /></a>
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/dm/@nestjs/common.svg" alt="NPM Downloads" /></a>
<a href="https://circleci.com/gh/nestjs/nest" target="_blank"><img src="https://img.shields.io/circleci/build/github/nestjs/nest/master" alt="CircleCI" /></a>
<a href="https://coveralls.io/github/nestjs/nest?branch=master" target="_blank"><img src="https://coveralls.io/repos/github/nestjs/nest/badge.svg?branch=master#9" alt="Coverage" /></a>
<a href="https://discord.gg/G7Qnnhy" target="_blank"><img src="https://img.shields.io/badge/discord-online-brightgreen.svg" alt="Discord"/></a>
<a href="https://opencollective.com/nest#backer" target="_blank"><img src="https://opencollective.com/nest/backers/badge.svg" alt="Backers on Open Collective" /></a>
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://opencollective.com/nest/sponsors/badge.svg" alt="Sponsors on Open Collective" /></a>
<a href="https://paypal.me/kamilmysliwiec" target="_blank"><img src="https://img.shields.io/badge/Donate-PayPal-ff3f59.svg" alt="Donate us"/></a>
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://img.shields.io/badge/Support%20us-Open%20Collective-41B883.svg" alt="Support us"></a>
<a href="https://twitter.com/nestframework" target="_blank"><img src="https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow" alt="Follow us on Twitter"></a>
</p>
<!--[![Backers on Open Collective](https://opencollective.com/nest/backers/badge.svg)](https://opencollective.com/nest#backer)
[![Sponsors on Open Collective](https://opencollective.com/nest/sponsors/badge.svg)](https://opencollective.com/nest#sponsor)-->

## Description

[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository.

## Project setup

```bash
$ yarn install
```

## Compile and run the project

```bash
# development
$ yarn run start

# watch mode
$ yarn run start:dev

# production mode
$ yarn run start:prod
```

## Run tests

```bash
# unit tests
$ yarn run test

# e2e tests
$ yarn run test:e2e

# test coverage
$ yarn run test:cov
```

## Deployment

When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information.

If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:

```bash
$ yarn install -g mau
$ mau deploy
```

With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure.

## Resources

Check out a few resources that may come in handy when working with NestJS:

- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework.
- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks.
- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).

## Support

Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).

## Stay in touch

- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec)
- Website - [https://nestjs.com](https://nestjs.com/)
- Twitter - [@nestframework](https://twitter.com/nestframework)

## License

Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE).
## To create migation
```npm run typeorm migration:create ./src/migrations/add-owner-to-token```
11 changes: 10 additions & 1 deletion packages/replica-server/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"build": "nest build",
"format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
"start": "nest start",
"prestart:dev": "npm run typeorm:migration:run",
"start:dev": "nest start --watch",
"start:debug": "nest start --debug --watch",
"start:prod": "node dist/main",
Expand All @@ -17,15 +18,23 @@
"test:watch": "jest --watch",
"test:cov": "jest --coverage",
"test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
"test:e2e": "jest --config ./test/jest-e2e.json"
"test:e2e": "jest --config ./test/jest-e2e.json",
"typeorm": "ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli.js",
"typeorm:migration:run": "typeorm -- migration:run -d --dataSource ./src/config/datasource.ts"
},
"dependencies": {
"@nestjs/bull": "^10.2.2",
"@nestjs/bullmq": "^10.2.2",
"@nestjs/common": "^10.0.0",
"@nestjs/config": "^3.3.0",
"@nestjs/core": "^10.0.0",
"@nestjs/platform-express": "^10.0.0",
"@nestjs/typeorm": "^10.0.2",
"@nft-open-marketplace/interface": "^0.2.1",
"bull": "^4.16.4",
"bullmq": "^5.25.4",
"class-transformer": "^0.5.1",
"class-validator": "^0.14.1",
"ethers": "^6.13.4",
"pg": "^8.13.1",
"pinata-web3": "^0.5.2",
Expand Down
22 changes: 0 additions & 22 deletions packages/replica-server/src/app.controller.spec.tss

This file was deleted.

18 changes: 0 additions & 18 deletions packages/replica-server/src/app.module.ts

This file was deleted.

14 changes: 4 additions & 10 deletions packages/replica-server/src/config/database.module.ts
Original file line number Diff line number Diff line change
@@ -1,22 +1,16 @@
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { getDbConfig } from './datasource';

@Module({
imports: [
TypeOrmModule.forRootAsync({
imports: [ConfigModule], // Make sure ConfigModule is imported
inject: [ConfigService],
useFactory: async (configService: ConfigService) => ({
type: 'postgres',
host: configService.get<string>('DATABASE_HOST'),
port: configService.get<number>('DATABASE_PORT'),
username: configService.get<string>('DATABASE_USERNAME'),
password: configService.get<string>('DATABASE_PASSWORD'),
database: configService.get<string>('DATABASE_NAME'),
entities: [__dirname + '/../**/*.entity{.ts,.js}'],
synchronize: true, // TODO set to false in production
}),
useFactory: async (configService: ConfigService) =>
getDbConfig(configService)
,
}),
],
})
Expand Down
34 changes: 34 additions & 0 deletions packages/replica-server/src/config/datasource.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import { DataSource } from 'typeorm';
import { ConfigService } from '@nestjs/config';
import { TypeOrmModuleOptions } from '@nestjs/typeorm';
import { PostgresConnectionOptions } from 'typeorm/driver/postgres/PostgresConnectionOptions';
import { QueueOptions } from 'bullmq';

const cs = new ConfigService();

export const getDbConfig = (configService: ConfigService): TypeOrmModuleOptions => {
return {
type: 'postgres',
host: configService.get<string>('DATABASE_HOST'),
port: configService.get<number>('DATABASE_PORT'),
username: configService.get<string>('DATABASE_USERNAME'),
password: configService.get<string>('DATABASE_PASSWORD'),
database: configService.get<string>('DATABASE_NAME'),
entities: [__dirname + '/../**/*.entity{.ts,.js}'],
synchronize: true, // TODO set to false in production
// TO generate intial migration:
// npm run typeorm -- migration:generate -d ./src/config/datasource.ts ./src/migrations/initial.ts
migrations: [__dirname + '/../migrations/*.entity{.ts,.js}']
}
}

export const AppDataSource = new DataSource(getDbConfig(cs) as PostgresConnectionOptions);

export const getRedisConfig = (configService: ConfigService): QueueOptions => {
return {
connection: {
host: configService.get("REDIS_HOST"),
port: configService.get("REDIS_PORT"),
},
}
}
17 changes: 17 additions & 0 deletions packages/replica-server/src/config/redis.module.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { BullModule } from '@nestjs/bullmq';
import { Module } from '@nestjs/common';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { getRedisConfig } from './datasource';

@Module({
imports: [
BullModule.forRootAsync({
imports: [ConfigModule],
inject: [ConfigService],
useFactory: async (configService: ConfigService) => getRedisConfig(configService),
}),
],
exports: [BullModule],
})
export class RedisModule {}

Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
import { Injectable, OnModuleInit } from '@nestjs/common';
import { ethers } from 'ethers';
import { ContractEventPayload, ethers, toNumber } from 'ethers';
import {
openMarketplaceNFTContractAbi,
} from '@nft-open-marketplace/interface';
import { readFileSync } from 'fs';
import { resolve } from 'path';
import { ConfigService } from '@nestjs/config';
import { TransferEventService } from 'src/nft/services/transfer-event.service';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { JobName, QueueName } from '../bus/consts';
import { TransferEventJob } from '../bus/types';

const contractsData = JSON.parse(
readFileSync(resolve('../../shared/contracts.deploy-data.json'), 'utf8'),
Expand All @@ -22,7 +25,8 @@ export class BlockchainListenerService implements OnModuleInit {

constructor(
private configService: ConfigService,
private transferEventService: TransferEventService,
@InjectQueue(QueueName.transferEvent) private transferEventQueue: Queue<TransferEventJob>,
// @InjectQueue('listing-queue') private listingQueue: Queue,
) {
// const wsProviderUrl = 'wss://mainnet.infura.io/ws/v3/YOUR_PROJECT_ID';
const wsProviderUrl = `ws://${this.configService.get(
Expand All @@ -46,15 +50,26 @@ export class BlockchainListenerService implements OnModuleInit {
}

private listenToEvents() {
console.log('Tran lsi')

this.contract.on('Transfer', (from, to, tokenId, event) => {
this.contract.on('Transfer', async (from: string, to: string, tokenId: BigInt, eventData: ContractEventPayload) => {
/*
TODO should be refactored using bus (rabbitMQ/kafka).
Saving will be moved to consumer.
Also requries sync worker for server downtime
*/
this.transferEventService.save(from, to, tokenId, event);
// this.transferEventService.save(from, to, tokenId, eventData);
console.log('publish', tokenId)
await this.transferEventQueue.add(JobName.Transfer, {
from,
to,
tokenId: parseInt(tokenId.toString()),
log: {
blockHash: eventData.log.blockHash,
blockNumber: eventData.log.blockNumber,
address: eventData.log.address,
transactionHash: eventData.log.transactionHash,
transactionIndex: eventData.log.transactionIndex,
}
});
});
}
}
6 changes: 6 additions & 0 deletions packages/replica-server/src/core/bus/consts.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
export enum JobName {
Transfer = 'transfer'
}
export enum QueueName {
transferEvent = 'transfer_event'
}
25 changes: 25 additions & 0 deletions packages/replica-server/src/core/bus/transfer-event.processor.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { TransferEventService } from '../nft/services/transfer-event.service';
import { TransferEventJob } from './types';
import { QueueName } from './consts';
import { Job } from 'bullmq';

@Processor(QueueName.transferEvent)
export class TransferEventConsumer extends WorkerHost {
constructor(
private transferEventService: TransferEventService,
) {
super();
}

async process(job: Job<TransferEventJob>, token?: string): Promise<any> {
console.log('job id->', job.id);
console.log('token', token);
try {
this.transferEventService.save(job.data);
} catch (e) {
console.error('gavno', e);
}
return { done: true };
}
}
Loading

0 comments on commit 43e6220

Please sign in to comment.