adds telemetry

[epinault]

This is an attempt to add some basic telemetry. This will be useful to then create an OTEL package to automatically get tracing from this telemetry setup.

I am happy to add tests as well but the setup is currently difficult in this project to run outside of CI. Is that the intent to not be able to run them locally? Is there way to make them run locally?

[brainlid]

Hi @epinault! The test should be easy to run locally. The main challenge is that some tests are "live", meaning they hit a server and test the responses. Those require an API key for the service being tested. I typically run the tests locally and focus on the live tests for the service I'm working with currently.

Otherwise, running mix test runs all the non-live tests locally. In general, I try to have most of the important parsing logic run using non-live tests.

[epinault]

I think because it require the API key and I don t have any, thats where the challenge was. The test_helper does a fetch_env! . I ll just put fake value and see what happens. Just feels we could make it a little easier so we dont even need them in test for local test?

[brainlid]

There's a file .envrc_template in the root of the project. Rename it to .envrc and if you're using something like direnv, then the dummy API keys get loaded for you.

If there's something else needed in the Testing section of the README, I'm happy to update it.

[epinault]

@brainlid thanks! that helped! I updated the README and added tests. what do you think about the MR?

[epinault]

@brainlid any chance we could merge this in?

[brainlid]

Hi @epinault!

I tried to update the PR, but failed and gave up after about 30 minutes. I'm guessing the PR/branch isn't setup to "Allow edits from maintainers", either that or I was just having a bad day.

It looks good! I like the telemetry module with the event docs and examples.

Here's what I was trying to do:

• Merged master in
• Fix merge errors
• Fix variable names in ChatMistralAI (openai -> mistralai)
• Run formatter

There are about 5 tests failing in test/chat_models/telemetry_test.exs. It appears that they are running as "live" tests and failing with bad API credentials, which makes me think the mocking isn't setup as expected for the tests?

Once we get this fixed up I'll be happy to merge it!

[epinault]

sounds good! I ll take a look this week and try to fix it up

[epinault]

rebased, fixed up the merge error, and the variable name. Also reformatted with 1.18

Looking at the 5 tests now

[epinault]

ok hopefully test are passing for you now too! I used a bit of "AI" to get the tests fixed up. 😅

[brainlid]

Awesome @epinault! Thanks! This is a valuable addition.
❤️💛💙💜

[epinault]

thank you for merging this! Now I just need to make an opentelemetry version of that for tracing 🫠