Log-App-Golang

Log Ingestor and Log Search App

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contact
5. Resources

About The Project

Log ingestor system that can efficiently handle vast volumes of log data, and offer a simple interface for querying this data using full-text search or specific field filters.

(back to top)

Built With

• Golang
• Gin
• MongoDB (Atlas)
• HTML (For UI)
• Docker

(back to top)

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

Need to have either docker or go installed.

Installation

Docker Based

1. Clone the repo

   git clone https://github.com/dyte-submissions/november-2023-hiring-pawarpranav83.git

2. Run docker-compose yaml file. It will build the image and run it on port :3000

   docker compose up

Golang Based

1. Clone the repo

   git clone https://github.com/dyte-submissions/november-2023-hiring-pawarpranav83.git

2. Run

   go build .

(back to top)

Usage

Log Ingestor

Perform a POST request with Log data on http://localhost:3000/

Note - The format should be as per the given sample, if any of the fields are nil, the server will give an error. This validation is performed using JSON and binding tags (provided by gin).

It would return an InsertedID, for successful insertion in the database.

(back to top)

Query Interface

HTML UI can be accessed with GET request on http://localhost:3000/

Uses an HTML Form with fields for searching, and on submit, makes a POST request to http://localhost:3000/search, which displays the docs obtained. One field for Full-Text Based Search - Text-Search.

Other fields act as filters.

Created an Aggregation Pipeline -

1. First Stage - Full-Text Based Search on all documents of a collection (created a search index using atlas search).

2. Second Stage - Matching / Filtering the documents obtained from the first stage based on the input filter parameters. Equal to operation, but for message we use the text-based search (created a text index for message), but can only be used when Text-Search field is empty (since MongoDB only allows text-based search on first stage).

3. Third Stage - Limit the number of documents to five.

Requirements

Log Ingestor

• Develop a mechanism to ingest logs in the provided format. Performed Validation using struct tags and the ShouldBindWithJSON method provided by Gin.
• Ensure scalability to handle high volumes of logs efficiently. Gin processes requests concurrently, the server can handle high volumes of logs.
• Mitigate potential bottlenecks such as I/O operations, database write speeds, etc. Since only text-based search indexes are added, no bottlenecks in write speeds, and since requests process concurrently so no blocking because of I/O operations.
• Make sure that the logs are ingested via an HTTP server, which runs on port 3000 by default.

Query Interface

• Offer a user interface (Web UI or CLI) for full-text search across logs. Implemented HTML Form for searching logs.
• Filters. Have added filters for all fields and can be used by specifying a value for a field in their corresponding input field.
• Aim for efficient and quick search results. Have search-based indexes that result in a quick text-based search.

Advance Features

• Implement search within specific date ranges. Using gte and lte operators in match stage for timestamp. Also timestamp is stored as value using time.Parse method in golang.
• Allow combining multiple filters. Multiple filters can be combined, if not null those fields are added in the match stage filters. Combined using AND operator.

(back to top)

Contact

Pranav Pawar - @pawarpranav83 - pawar.pranav83@gmail.com

Phone Number - 8104273543

(back to top)

Resources

• MongoDB Go Driver Docs - https://www.mongodb.com/docs/drivers/go/current/
• Gin Docs - https://github.com/gin-gonic/gin/blob/v1.9.1/docs/doc.md

(back to top)