json-transform is a CLI tool and microservice that allows you to perform automated batch operations on JSON files using OpenAI (ChatGPT), Google Gemini and other AI LLMs.
Ensure you have Go installed on your system.
-
Clone the repository:
git clone https://github.com/matt-dunleavy/json-transform.git cd json-transform
-
Install dependencies:
go mod tidy
-
Build the binary:
go build -o json-transform
-
Move it to a directory in your PATH:
sudo mv json-transform /usr/local/bin/ # For Linux/MacOS move json-transform C:\Windows\System32 # For Windows
json-transform [command]
Perform batch operations on JSON files with the batch
command by providing the directory containing the source json files, the desired output directory, and the desired batch operations.
json-transform batch -i <input_directory> -o <output_directory> -p <operation> -k <api_key> -f <prompt_file> -s <api_service> -m <model>
-i, --input
(required): Input directory containing JSON files.-o, --output
(required): Output directory for processed JSON files.-p, --operation
(required): Batch operation to perform (merge, split, filter, api-process).-k, --api-key
(required): API key for external API.-f, --prompt
(required): File containing the prompt for API processing.-s, --api-service
: API service to use (chatgpt or gemini). Defaults tochatgpt
.-m, --model
: AI model to use.
json-transform batch -i ./input -o ./output
Transform JSON files using specified rules.
json-transform transform -i <input_file> -o <output_file> -r <rules_file>
Flags:
-i, --input
(required): Input JSON file.-o, --output
(required): Output JSON file.-r, --rules
(required): Transformation rules file.
json-transform transform -i input.json -o output.json -r rules.json
Validate JSON files against a schema.
json-transform validate -i <input_file> -s <schema_file>
Flags:
-i, --input
(required): Input JSON file.-s, --schema
(required): JSON schema file.
Example:
json-transform validate -i input.json -s schema.json
To get help for any command, you can use the --help
flag.
json-transform --help
or
json-transform [command] --help
The json-transform
tool leverages AI APIs (e.g. ChatGPT, Google Gemini, etc.) to perform advanced operations on JSON files.
-
Setup API Keys:
- Ensure you have a valid API key for the chosen AI service (ChatGPT or Gemini).
- Keep your API key secure and do not expose it in your code or public repositories.
-
Prepare the Prompt File:
- Create a text file containing the prompt that will be used by the AI service. This prompt should include instructions on what you want the AI to do with the input JSON content.
- Use placeholders like
{{input}}
to denote where the content from the JSON file should be inserted in the prompt.
Run the Batch Command:
- Use the
batch
command with the appropriate flags to process JSON files using the AI service. Specify the input directory, output directory, operation (api-process
), API key, prompt file, API service (chatgpt
orgemini
), and the model if applicable.
json-transform batch -i <input_directory> -o <output_directory> -p api-process -k <api_key> -f <prompt_file> -s <api_service> -m <model>
Example:
json-transform batch -i ./input -o ./output -p api-process -k your_api_key -f prompt.txt -s chatgpt -m text-davinci-003
Review the Output:
- The processed JSON files will be saved in the specified output directory.
- Responses from the AI service will be logged to the console and written to the output files.
- Prompt Design:
- Craft clear and concise prompts to guide the AI in performing the desired task.
- Test and iterate on your prompts to improve the accuracy and relevance of the AI's responses.
- Input Data Quality:
- Ensure the input JSON files are well-structured and contain the necessary data for the AI to process.
- Clean and preprocess your data to avoid errors during processing.
- Model Selection:
- Choose the appropriate AI model based on the complexity and nature of the task. For example, use
text-davinci-003
for more complex and nuanced tasks with ChatGPT.
- Choose the appropriate AI model based on the complexity and nature of the task. For example, use
- Error Handling:
- Implement robust error handling to manage issues such as network errors, API rate limits, and invalid input data.
- Log errors for troubleshooting and to understand any issues with the AI processing.
- Security:
- Keep your API key secure and avoid hardcoding it in scripts or sharing it publicly.
- Follow best practices for managing sensitive information, such as using environment variables to store API keys.
- Usage Limits:
- Be aware of the API usage limits and quotas for your AI service. Monitor your usage to avoid exceeding these limits.
- Consider implementing rate limiting in your application to stay within the allowed usage limits.
- Scalability:
- For large-scale processing, consider batching requests and using asynchronous processing to handle high volumes of JSON files efficiently.
- Optimize the performance of your application to handle the load without affecting the AI service's response time.
By following these instructions and best practices, you can effectively leverage the AI capabilities of the json-transform
tool to perform advanced operations on your JSON files.
Utilize the AI API to summarize long text documents stored in JSON files. This is particularly useful for processing large amounts of textual data such as reports, articles, or meeting transcripts.
json-transform batch -i ./documents -o ./summaries -p api-process -k your_api_key -f summarize_prompt.txt -s chatgpt -m text-davinci-003
Prompt: (e.g. summarize_prompt.txt
):
Summarize the following text:
{{input}}
Use the AI API to extract entities (like names, dates, locations, etc.) from JSON files containing log data. This can help in parsing and structuring unstructured log data for further analysis.
json-transform batch -i ./logs -o ./structured_logs -p api-process -k your_api_key -f entity_extraction_prompt.txt -s chatgpt -m text-davinci-003
Prompt (e.g. entity_extraction_prompt.txt
):
Extract all the entities (names, dates, locations, etc.) from the following log entry:
{{input}}
Integrate the AI API to generate code snippets or scripts from JSON files containing specifications. This can be useful for automating code generation tasks based on defined requirements.
json-transform batch -i ./specifications -o ./generated_code -p api-process -k your_api_key -f code_generation_prompt.txt -s chatgpt -m text-davinci-003
Prompt (e.g. code_generation_prompt.txt
):
Generate a Python script based on the following specification:
{{input}}
These examples illustrate how the AI API integration can be utilized to enhance the functionality of the json-transform
tool, providing powerful automation capabilities for a variety of tasks.
Leverage the AI API to translate JSON files containing text in one language to another language. This is useful for global businesses needing to process multi-language documents.
json-transform batch -i ./spanish_texts -o ./english_texts -p api-process -k your_api_key -f translate_prompt.txt -s chatgpt -m text-davinci-003
Prompt (e.g. translate_prompt.txt
):
Translate the following text from Spanish to English:
{{input}}
Use the AI API to perform sentiment analysis on JSON files containing customer feedback or reviews. This can help businesses understand customer sentiments and improve their products or services.
json-transform batch -i ./feedback -o ./processed -p api-process -k your_api_key -f sentiment_prompt.txt -s chatgpt -m text-davinci-003
Prompt: (e.g. sentiment_prompt.txt
)
Analyze the following customer feedback and determine the sentiment (positive, negative, neutral):
{{input}}
json-transform/
├── cmd/
│ ├── root.go
│ ├── batch.go
│ ├── transform.go
│ ├── validate.go
├── config/
│ ├── config.go
├── internal/
│ ├── batch/
│ │ ├── processor.go
│ │ └── processor_test.go
│ ├── transform/
│ │ ├── transformer.go
│ │ └── transformer_test.go
│ ├── validate/
│ │ ├── validator.go
│ │ └── validator_test.go
│ ├── api/
│ │ ├── api.go
│ │ └── api_test.go
│ ├── utils/
│ │ ├── json.go
│ │ └── json_test.go
├── scripts/
│ ├── run.sh
│ ├── build.sh
├── testdata/
│ ├── sample1.json
│ ├── sample2.json
├── Dockerfile
├── .gitignore
├── go.mod
├── go.sum
├── main.go
└── README.md
Contributions are welcome! Please open an issue or submit a pull request on GitHub.
This package is distributed under the MIT License.