I2C library discovery tool

[fhunleth]

Over the past few years, I've learned that new users have a hard time finding device libraries so they sometimes re-implement support for a device (or give up). Good libraries also go unused due to naming being difficult (not I2C, but how do you name a library for WS2812 LEDs so everyone finds it? Name it neopixel? or what if dotstar support is added, then is it called "leds". Or just name it the LED part number).

It seems like Circuits.I2C.detect_devices/1 (or a new function) could help. Hex.pm has an API to query for packages. The query might be all libraries that depend on circuits_i2c and have an I2C address in their metadata. We'd have to publicize and work with library authors to add appropriate metadata to their packages.

[mattludwigs]

I like this idea! I am thinking I should be a different function primarily for clarity and if we want to add more advanced querying to the function we are not slamming that into the same function that provides a list of i2c devices that are on the bus.

I can work on this later this week, as I need to get the first pass of the website in a PR first.

[mattludwigs]

So I have a basic prototype CLI tool that can search hex by dependency and print out the packages that use that dependency.

However, I saw a conversation involving searching by nerves_system_br in the Elixir slack channel. I am not sure if the tooling for this is best as a CLI, general hex API library that others can use, or a one-off thing that if some project wants they just reimplement it.

I personally like having the CLI tool, but I can also see a more general hex api library that can be shared in different projects.

[fvasquez]

@mattludwigs I think your hex-search CLI tool is a fantastic idea. I can finish that tool for you if you are no longer interested in pursuing it. Right now the list of "What uses Elixir Circuits?" on the website is where most developers will go to search for device libraries.

[mattludwigs]

Thanks!

I stopped working on as we never decided if it should live in the circuits code so that one would have to get the dep and use iex to run a function to list this out, or if it should live outside the elixir code so that a user can search hex from their terminal. Or even something else.

I wrote mine to work more generically outside of the context of elixir circuits and to work as a CLI tool. Which also means it's in a different language than Elixir, which is less shareable to other Elixir projects. I know personally, I would use a tool like that. If that is desirable I am not sure if that would live in elixir circuits though. Maybe it can be be made into a mix archive, so the tool can be installed and ran as a mix task?

Maybe the answer is both?

Also, if there is a way to make the packages more searchable on the webpage I am open to thoughts on that. @fhunleth I would like to get your thoughts about this.

[fhunleth]

This seems really useful. Is there anything to look at? If the question is whether to integrate the tool in Circuits.I2C, then I think that decision is made after there's some experience using and maintaining it. When the tool is available, I think we'll definitely want to put a link to it here to help people find it.

[mattludwigs]

I guess my question is:

Should it be Elixir based CLI via mix task or another language? Is CLI a good solution?

I have some really rough prototype code I can share but its more generalized then circuits.... and it's not in Elixir. 🦀

[fvasquez]

A general purpose CLI tool that could search for hex packages by dependency would benefit the entire Elixir community. I think it should be a mix task since anyone looking for hex packages is probably using mix and thereby already has it installed. CLI is a great solution. I would definitely reach for a CLI tool before I start cobbling together raw queries against the hex API. I looked at the mix documentation and there is nothing like a mix hex.dependson task. There should be.

[mattludwigs]

I think that is a really good point. From conversations, I think that there are few things to do that would both accomplish shared code, plus the specific goals for circuits (and other libraries).

1. A hex API client if that does not exist yet. The scope of this can probably just searching the depends on to start with, I am not sure if there is need a for a full feature API client upfront.
2. The mix task CLI we have talked about above.

For those points, I don't think circuits should own those repos, but it can use the first one. Also, shared code and the CLI tooling are both made available.

3. Circuit projects can pull the first dep and provide a nice interface for searching for only things those repos care about.

Interested to hear other thoughts on this. I think the goals are realistic and meet the desired goals discussed throughout this thread.

[fvasquez]

I started working on task number 1 yesterday. You can find the repo for my Hex API client here. I plan to use HTTPoison to query Hex.pm. I'll let you know when I have it working so Circuits can link to it.

[fvasquez]

I went through the Hex API spec on Apiary and it turns out that "dependents" is not an attribute of the "package" JSON schema. Even retrieving the list of "dependencies" for a package is a bit roundabout using the Hex API. The "package" response includes a list of "releases" rather than dependencies. You then need to issue another request to the Hex API for a "release" by package "name" and release "version" to get the list of "dependencies" for a particular package release.

From what I can tell, the Hex.pm website relies on an Ecto query to a database to fill in the "Dependents(n):" section that appears on each package web page.

Excerpt from hexpm/lib/hexpm_web/controllers/package_controller.ex

    dependants =
      Packages.search(
        repositories,
        1,
        20,
        "depends:#{package.name}",
        :recent_downloads,
        [:name, :repository_id]
      )

    dependants_count = Packages.count(repositories, "depends:#{package.name}")

    render(
      conn,
      "show.html",
      [
        title: package.name,
        description: package.meta.description,
        container: "container package-view",
        canonical_url: Routes.package_url(conn, :show, package),
        package: package,
        releases: releases,
        current_release: release,
        downloads: downloads,
        owners: owners,
        dependants: dependants,
        dependants_count: dependants_count
      ] ++ docs_assigns
    )

Excerpt from hexpm/lib/hexpm/repository/package.ex

  defp search_param("depends", search, query) do
    from(
      p in query,
      join: pd in Hexpm.Repository.PackageDependant,
      on: p.id == pd.dependant_id,
      where: pd.name == ^search
    )
  end

Based on the above, I don't think the Hex.pm website fetches "depends" from the Hex API server which is unfortunate.

@mattludwigs I am now very curious now how your prototype hex-search CLI tool was able to fetch "depends". Did you just scrape it from the Hex.pm web page for a given package?

[mattludwigs]

I haven't looked too closely at their documentation and it's been a long time so I don't remember what lead me to using this endpoint:

https://hex.pm/api/packages?search=depends%3Acircuits_i2c

Which in my prototype CLI I interpolate the url like so (it's not in Elixir):

let url = format!("https://hex.pm/api/packages?search=depends%3A{}", dep);

Where dep is what is passed into the CLI as the argument.

[fvasquez]

Thank you @mattludwigs! That API search request works beautifully.

[fvasquez]

So my hexquery CLI tool is done for now. You can find it here. All it does is fetch depends for a given package and filter the results by an optional search string like so.

body
|> Jason.decode!
|> Enum.filter(fn(package) ->
      if search_string = context[:containing] do
        description = package["meta"]["description"]
        String.contains?(description, search_string)
      else
        true
      end
    end)
|> Enum.map(fn(package) -> IO.puts package["name"] end)

If there were I2C device addresses included somewhere in the Hex package descriptions a hexquery user could filter circuits_i2c's dependents by an I2C address (eg. "0x60" would match the atecc508a package). I couldn't find any such addresses in the package descriptions so users should have better luck searching for strings like part model number instead.

Let me know if you have any feature requests. I'd like to extend this project into a more general purpose Hex API client and add more useful functionality like search for packages by owner. Let's see if there is any interest first.