Skip to content

Commit

Permalink
Merge pull request #4 from Scale3-Labs/karthik/update-readme
Browse files Browse the repository at this point in the history
Update Readme
  • Loading branch information
karthikscale3 authored Apr 1, 2024
2 parents 6b68a58 + 3db7ee1 commit 47981a0
Show file tree
Hide file tree
Showing 12 changed files with 173 additions and 48 deletions.
129 changes: 128 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,128 @@
<!-- Readme update -->
# [Langtrace](https://www.langtrace.ai)


## Open Source & Open Telemetry(OTEL) Observability for LLM applications.

![Static Badge](https://img.shields.io/badge/License-AGPL--3.0-blue) ![Static Badge](https://img.shields.io/badge/npm_@langtrase/typescript--sdk-1.2.9-green) ![Static Badge](https://img.shields.io/badge/pip_langtrace--python--sdk-1.2.8-green) ![Static Badge](https://img.shields.io/badge/Development_status-Active-green)

---

Langtrace is an open source observability software which lets you capture, debug and analyze traces and metrics from all your applications that leverages LLM APIs, Vector Databases and LLM based Frameworks.

## Open Telemetry Support

The traces generated by Langtrace adhere to [Open Telemetry Standards(OTEL)](https://opentelemetry.io/docs/concepts/signals/traces/). We are developing [semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/) for the traces generated by this project. You can checkout the current definitions in [this repository](https://github.com/Scale3-Labs/langtrace-trace-attributes/tree/main/schemas). Note: This is an ongoing development and we encourage you to get involved and welcome your feedback.

---

## Getting Started

### Langtrace Cloud ☁️
To use the managed SaaS version of Langtrace, follow the steps below:
1. Sign up by going to [this link](www.langtrace.ai).
2. Create a new Project after signing up. Projects are containers for storing traces and metrics generated by your application. If you have only one application, creating 1 project will do.
3. Generate an API key by going inside the project.
4. In your application, install the Langtrace SDK and initialize it with the API key you generated in the step 3.
5. The code for installing and setting up the SDK is shown below:


If your application is built using **typescript/javascript**:
```
npm i @langtrase/typescript-sdk
```

```
import { init } from '@langtrace-init/init';
init({ api_key: process.env.LANGTRACE_API_KEY });
```


If your application is built using **python**:
```
pip install langtrace-python-sdk
```

```
from langtrace_python_sdk import langtrace
langtrace.init(api_key=process.env.LANGTRACE_API_KEY)
```

### Langtrace self hosted

Langtrace UI is built using [NextJS](https://nextjs.org/). To self-host and use langtrace, you can use our Docker container:
```
# Clone the repository
git clone git@github.com:Scale3-Labs/langtrace.git
cd langtrace
# ⭐ Don't forget to star this repository ⭐
# Run the application and the databases locally
docker compose up
```

Install the langtrace SDK in your application by following the same instructions under the Langtrace Cloud section above for sending traces to your self hosted setup.

---
## SDK Repositories

- [Langtrace Typescript SDK](https://github.com/Scale3-Labs/langtrace-typescript-sdk)
- [Langtrace Python SDK](https://github.com/Scale3-Labs/langtrace-python-sdk)
- [Semantic Span Attributes](https://github.com/Scale3-Labs/langtrace-trace-attributes)

---
## Supported integrations

Langtrace automatically captures traces from the following vendors:

| Vendor | Type | Typescript SDK | Python SDK
| ------ | ------ | ------ | ------ |
| OpenAI | LLM | :white_check_mark: | :white_check_mark: |
| Anthropic | LLM | :white_check_mark: | :white_check_mark: |
| Azure OpenAI | LLM | :white_check_mark: | :white_check_mark: |
| Langchain | Framework | :x: | :white_check_mark: |
| LlamaIndex | Framework | :white_check_mark: | :white_check_mark: |
| Pinecone | Vector Database | :white_check_mark: | :white_check_mark: |
| ChromaDB | Vector Database | :white_check_mark: | :white_check_mark: |

---
## Feature Requests and Issues
- To request for features, head over [here to start a discussion](https://github.com/Scale3-Labs/langtrace/discussions/categories/feature-requests).
- To raise an issue, head over [here and create an issue](https://github.com/Scale3-Labs/langtrace/issues).


---
## Contributions

We welcome contributions to this project. To get started, fork this repository and start developing. To get involved, join our Slack workspace.

---
## Security

To report security vulnerabilites, email us at security@scale3labs.com. You can read more on security [here](https://github.com/Scale3-Labs/langtrace/blob/development/SECURITY.md).

---
## License

- Langtrace application(this repository) is [licensed](https://github.com/Scale3-Labs/langtrace/blob/development/LICENSE) under the AGPL 3.0 License. You can read about this license [here](https://www.gnu.org/licenses/agpl-3.0.en.html).
- Langtrace SDKs are licensed under the Apache 2.0 License. You can read about this license [here](https://www.apache.org/licenses/LICENSE-2.0).

---
## Frequently Asked Questions
**1. Can I self host and run Langtrace in my own cloud?**
Yes, you can absolutely do that. Follow the self hosting setup instructions laid out above.

**2. What is the pricing for Langtrace cloud?**
Currently, we are not charging anything for Langtrace cloud and we are primarily looking for feedback so we can continue to improve the project. We will inform our users when we decide to monetize it.

**3. What is the tech stack of Langtrace?**
Langtrace uses NextJS for the frontend and APIs. It uses PostgresDB as a metadata store and Clickhouse DB for storing spans, metrics, logs and traces.

**4. Can I contribute to this project?**
Absolutely! We love developers and welcome contributions. Get involved early by joining our slack workspace.

**5. What skillset is required to contribute to this project?**
Programming Languages: Typescript and Python.
Framework knowledge: NextJS.
Database: Postgres and Prisma ORM.
Nice to haves: Opentelemetry instrumentation framework, experience with distributed tracing.
22 changes: 8 additions & 14 deletions app/api/evaluation/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,16 +119,13 @@ export async function GET(req: NextRequest) {
});

if (!evaluations) {
return NextResponse.json(
{
error: "No evaluation found",
},
{ status: 404 }
);
return NextResponse.json({
evaluations: [],
});
}

return NextResponse.json({
evaluations,
evaluations: [evaluations],
});
}

Expand All @@ -140,16 +137,13 @@ export async function GET(req: NextRequest) {
});

if (!evaluations) {
return NextResponse.json(
{
error: "No evaluations found",
},
{ status: 404 }
);
return NextResponse.json({
evaluations: [],
});
}

return NextResponse.json({
evaluations,
evaluations: [evaluations],
});
}

Expand Down
4 changes: 2 additions & 2 deletions components/charts/model-accuracy-chart.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"use client";

import { AreaChart } from "@tremor/react";
import { BarChart } from "@tremor/react";
import { useQuery } from "react-query";
import { Info } from "../shared/info";

Expand Down Expand Up @@ -90,7 +90,7 @@ export function ModelAccuracyChart({ projectId }: { projectId: string }) {
{r.model} Overall Accuracy: {r.overallAccuracy.toFixed(2)}%
</p>
))}
<AreaChart
<BarChart
className="mt-2 h-72"
data={sortedArray}
index="date"
Expand Down
6 changes: 3 additions & 3 deletions components/charts/token-chart.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"use client";

import { AreaChart } from "@tremor/react";
import { BarChart } from "@tremor/react";
import { useQuery } from "react-query";

export function TokenChart({ projectId }: { projectId: string }) {
Expand Down Expand Up @@ -36,7 +36,7 @@ export function TokenChart({ projectId }: { projectId: string }) {
Total Tokens: {fetchMetricsUsageToken.data.totalTokens || 0}
</p>
</div>
<AreaChart
<BarChart
className="mt-2 h-72"
data={fetchMetricsUsageToken.data?.usage?.map((data: any) => ({
date: data.date,
Expand Down Expand Up @@ -94,7 +94,7 @@ export function CostChart({ projectId }: { projectId: string }) {
Total Cost: ${fetchMetricsUsageCost.data?.total?.toFixed(6) || 0}
</p>
</div>
<AreaChart
<BarChart
className="mt-2 h-72"
data={fetchMetricsUsageCost.data?.cost?.map((data: any) => ({
date: data?.date,
Expand Down
11 changes: 7 additions & 4 deletions components/charts/trace-chart.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"use client";

import { AreaChart } from "@tremor/react";
import { BarChart } from "@tremor/react";
import { useQuery } from "react-query";

export function TraceSpanChart({ projectId }: { projectId: string }) {
Expand Down Expand Up @@ -46,7 +46,10 @@ export function TraceSpanChart({ projectId }: { projectId: string }) {
const data = traceData?.map((trace: any, index: number) => {
return {
...trace,
"Span Count": spanData && spanData.length > 0 && spanData[index] !== undefined ? spanData[index]["Span Count"] : 0,
"Span Count":
spanData && spanData.length > 0 && spanData[index] !== undefined
? spanData[index]["Span Count"]
: 0,
};
});

Expand All @@ -67,7 +70,7 @@ export function TraceSpanChart({ projectId }: { projectId: string }) {
Spans are individual events that represent a single operation.
</p>
</div>
<AreaChart
<BarChart
className="mt-2 h-72"
data={data}
index="date"
Expand Down Expand Up @@ -113,7 +116,7 @@ export function SpanChart({ projectId }: { projectId: string }) {
Spans are individual events that represent a single operation.
</p>
</div>
<AreaChart
<BarChart
className="mt-2 h-72"
data={fetchMetricsUsageSpan.data.spans.map((data: any) => ({
date: data.date,
Expand Down
4 changes: 2 additions & 2 deletions components/project/dataset/create-data.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ export function CreateData({
const schema = z.object({
input: z.string().min(2, "Too short").max(200, "Too long"),
output: z.string().min(2, "Too short").max(2000, "Too long"),
note: z.string().min(2, "Too short").max(25, "Too long").optional(),
note: z.string().max(25, "Too long").optional(),
});
const CreateDataForm = useForm({
resolver: zodResolver(schema),
Expand Down Expand Up @@ -186,7 +186,7 @@ export function CreatePrompt({
const [busy, setBusy] = useState<boolean>(false);
const schema = z.object({
value: z.string().min(2, "Too short").max(2000, "Too long"),
note: z.string().min(2, "Too short").max(25, "Too long").optional(),
note: z.string().max(25, "Too long").optional(),
});
const CreatePromptsetForm = useForm({
resolver: zodResolver(schema),
Expand Down
2 changes: 2 additions & 0 deletions components/project/dataset/create.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ export function CreateDataset({
await queryClient.invalidateQueries(
"fetch-datasets-stats-query"
);
await queryClient.invalidateQueries("fetch-datasets-query");
toast("Dataset created!", {
description: "Your dataset has been created.",
});
Expand Down Expand Up @@ -204,6 +205,7 @@ export function CreatePromptset({
await queryClient.invalidateQueries(
"fetch-promptsets-stats-query"
);
await queryClient.invalidateQueries("fetch-promptsets-query");
toast("Promptset created!", {
description: "Your promptset has been created.",
});
Expand Down
4 changes: 2 additions & 2 deletions components/project/dataset/edit-data.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ export function EditData({
const schema = z.object({
input: z.string().min(2, "Too short").max(200, "Too long"),
output: z.string().min(2, "Too short").max(2000, "Too long"),
note: z.string().min(2, "Too short").max(25, "Too long").optional(),
note: z.string().max(25, "Too long").optional(),
});
const EditDataForm = useForm({
resolver: zodResolver(schema),
Expand Down Expand Up @@ -286,7 +286,7 @@ export function EditPrompt({
const [busy, setBusy] = useState<boolean>(false);
const schema = z.object({
value: z.string().min(2, "Too short").max(2000, "Too long"),
note: z.string().min(2, "Too short").max(25, "Too long").optional(),
note: z.string().max(25, "Too long").optional(),
});
const EditPromptSetForm = useForm({
resolver: zodResolver(schema),
Expand Down
15 changes: 9 additions & 6 deletions components/project/eval/eval.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,11 @@ const EvalRow = ({
queryFn: async () => {
const response = await fetch(`/api/evaluation?spanId=${prompt.span_id}`);
const result = await response.json();
setEvaluation(result.evaluations);
setScore(result.evaluations?.score || -100);
console.log(result);
setEvaluation(result.evaluations.length > 0 ? result.evaluations[0] : {});
setScore(
result.evaluations.length > 0 ? result.evaluations[0].score : -100
);
return result;
},
});
Expand Down Expand Up @@ -185,9 +188,9 @@ const EvalRow = ({

// score evaluation
const evaluateSpan = async (newScore: number) => {
if (score === -100) {
if (!evaluation?.id) {
// Evaluate
fetch("/api/evaluation", {
await fetch("/api/evaluation", {
method: "POST",
headers: {
"Content-Type": "application/json",
Expand All @@ -203,7 +206,7 @@ const EvalRow = ({
}),
});
} else {
fetch("/api/evaluation", {
await fetch("/api/evaluation", {
method: "PUT",
headers: {
"Content-Type": "application/json",
Expand All @@ -216,7 +219,7 @@ const EvalRow = ({
}

// Invalidate the evaluations query to refetch the updated evaluations
queryClient.invalidateQueries("fetch-evaluations-query");
queryClient.invalidateQueries(`fetch-evaluation-query-${prompt.span_id}`);
};

return (
Expand Down
20 changes: 8 additions & 12 deletions components/project/eval/prompts.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ import { useParams } from "next/navigation";
import { useState } from "react";
import Markdown from "react-markdown";
import { useQuery } from "react-query";
import SetupInstructions from "../../shared/setup-instructions";
import { Button } from "../../ui/button";
import { Separator } from "../../ui/separator";

Expand Down Expand Up @@ -90,17 +89,14 @@ export default function Prompts({ email }: { email: string }) {
/>
);
})}
{!fetchPrompts.isLoading &&
fetchPrompts.data &&
!fetchPrompts.data?.prompts?.result && (
<div className="flex flex-col gap-3 items-center justify-center p-4">
<p className="text-muted-foreground text-sm mb-3">
No prompts available. Get started by setting up Langtrace in
your application.
</p>
<SetupInstructions project_id={project_id} />
</div>
)}
{dedupedPrompts?.length === 0 && (
<div className="flex flex-col gap-3 items-center justify-center p-4">
<p className="text-muted-foreground font-semibold text-md mb-3">
No prompts available. Use the system role with your LLM API
calls to capture prompts automatically.
</p>
</div>
)}
</div>
</div>
);
Expand Down
4 changes: 2 additions & 2 deletions components/shared/nav.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ const ProjectNavLinks = (id: string) => [
href: `/project/${id}/traces`,
},
{
name: "Eval",
href: `/project/${id}/eval`,
name: "Evaluations",
href: `/project/${id}/evaluations`,
},
{
name: "Datasets",
Expand Down

0 comments on commit 47981a0

Please sign in to comment.