How can we make Shopify Ruby ecosystem better?

[vfonic]

Issue summary

This project has seen better days. So did ShopifyApp gem.

1. Docs site markdown is broken

https://shopify.github.io/shopify-api-ruby/usage/oauth.html

2. Docs are outdated

There's one week old PR about updating docs that still hasn't been merged:
#1172
Does it really take a week to get this merged?
BTW this has been reported more than a month ago:
#1163
Does it take 40 days (and counting) for a documentation fix?
2 commits, 4 documentation files changed, 19 lines of docs added and 81 lines of docs removed.
This takes 40+ days.
Shopify has truly abandoned their Ruby developers community.

In v13 of this gem there's no ShopifyAPI::Auth::SessionStorage.
This is being referenced in several places in documentation, even in README:

• https://github.com/Shopify/shopify-api-ruby/blob/main/README.md#breaking-change-notice-for-version-1000
• https://github.com/Shopify/shopify-api-ruby/blob/main/docs/usage/session_storage.md
• https://github.com/Shopify/shopify-api-ruby/blob/94591589ff57d8dcb3f37a97f70620766bb92ad6/docs/issues.md#notes-on-session-handling

There are also several issues about this opened.

3. Authentication documentation

I'm trying to figure out a simple thing of authenticating/starting an offline session in my Rails app.

It was done like this before:

session = ShopifyAPI::Session.new(domain: shop.shopify_domain, token: shop.shopify_token, api_version: api_version)
ShopifyAPI::Base.activate_session(session)

This no longer works.

I went to the documentation website:
https://shopify.github.io/shopify-api-ruby/getting_started.html
...and found nothing.

I found some info here on how to do online session:
https://github.com/Shopify/shopify-api-ruby/blob/main/docs/usage/rest.md

I found this piece of code here:
https://shopify.dev/docs/api/admin-rest

session = ShopifyAPI::Utils::SessionUtils.load_current_session(
  auth_header: auth_header,
  cookies: cookies,
  is_online: is_online
)
client = ShopifyAPI::Clients::Rest::Admin.new(
  session: session
)
response = client.get(path: 'shop')

I don't know how I can convert this for offline use?

I finally figured out how to achieve this by reading ShopifyApp and Shopify API gems source code:

session = ShopifyAPI::Auth::Session.new(shop: shop.shopify_domain, access_token: shop.shopify_token)
ShopifyAPI::Context.activate_session(session)

Seems like with all the unnecessary breaking changes, you also managed to rename domain: keyword argument to shop:, which is first unclear, but also wrong, because in many Rails apps shop represents an instance of a Shop ActiveRecord model.

4. Docs don't mention ShopifyApp gem

It's sad to see that on this page there's no reference to ShopifyApp gem, only to Shopify API gem:
https://shopify.dev/docs/api/admin-rest

I understand I'm on a part of documentation about REST API, but I could barely find any reference to ShopifyApp gem anywhere on the shopify.dev:
https://shopify.dev/docs/apps/auth/oauth/update/ruby#step-1-update-the-shopify_app-gem
Is this the best way for a new developer to find out that there's a ShopifyApp gem?

Here's another good candidate where you could mention ShopifyApp:
https://shopify.dev/docs/apps/tools/api-libraries

5. There are hundreds of issues never answered and automatically closed by a bot

This is inhuman and a slap in the face to anyone who took their time to report an issue and/or create a PR. If you cannot keep track of open issues, at least disable the stale bot.

https://github.com/Shopify/shopify-api-ruby/issues?q=is%3Aissue+is%3Aclosed

This tragedy has turned into a comedy:
Why Shopify is not actively taking more care of these issues? #986
People are fighting with stale bot. It is sad and hilarious at the same time.

Yeah, tell me about it. In the end my business partner and I ended just writing our own ecommerce platform.

#969 (comment)
😂

6. There are bugs being reported that seem quite critical to Ruby/Rails fundamental functionality

This seems rather critical:
ShopifyAPI::Rest::Base responds_to everything #1171

ShopifyAPI::Order.new.present? raises an error.

Or how about this one:
Updating assets is broken in v13.0.0 #1164

7. Newer versions are more concerned about rewrites than keeping the existing functionality, newer versions have less functionality than older versions

GraphQL Support? #1168
Updating assets is broken in v13.0.0 #1164
@nelsonwittwer Can you elaborate a bit more on how session storage is used for a pure api client? If session storage is removed from Context.setup, how would it get defined without shopify_app? What role does it play in this scenario?
Breaking change notice for version 10.0.0 GREATLY underrepresented #1103

8. New Sorbet type system makes error messages useless

ShopifyAPI::Webhooks::Registry.add_registration occurred TypeError in fields parameter #1147
REST API Shop resource definitions TypeError #1145
Upgrading from 9.5.1 to 10+ adds a sorbet dependency that affects unit tests unrelated to Shopify #1109 - this issue talks about how upgrading Shopify API gem breaks unrelated tests. This is the worst kind of behavior that can happen for a new version of gem that you're being forced to upgrade to. And the suggested fix is to install more tooling. Which is not a fix.
Accessing the taxes_included property on a ShopifyAPI::Shop object returns a Sorbet TypeError #1170
Accessing the arguments property on a ShopifyAPI::Event object returns a Sorbet TypeError #1152

There are two pages of issues if you search for typeerror:
https://github.com/Shopify/shopify-api-ruby/issues?q=typeerror

9. Shopify forces developers to constantly upgrade to their latest buggy, undocumented, and with-many-breaking-changes gem/libraries versions

Remember the App Bridge upgrading fiasco? Oh, I 'member: [Docs] Documentation still refers to app bridge v1.x #71
We were asked to upgrade to App Bridge 2.x because our apps will be delisted, but the documentation on how to upgrade/switch a Rails app to App Bridge was non-existent.
Shopify API gem v10 introduced many breaking changes and we're constantly forced to keep upgrading to the latest gem versions or our apps will be delisted from the App Store because we're missing the functionality X.

It would be great to release breaking changes as a major version bump, instead of minor Shopify/shopify_app#875

FWIW, we've decided to just use HTTParty directly to the REST API rather than migrate to v10+ of this gem.

#1103 (comment)

What can be done?

Please help me on this one, I'd love to hear more opinions/suggestions how we can make this better.
Here are some of my suggestions:

1. Dedicate more resources internally to Ruby gems

2. Don't make so many breaking changes in: a) Shopify REST/GraphQL APIs, authentication/session management, etc. and b) in ShopifyApp and Shopify API gems. Don't do things like removing session storage from this gem. Just leave it. There are tons of other functionalities you could add instead.

3. Announce a cash bounty and pay developers willing to work on Shopify's open source projects

4. This goes well with the previous point: make a frequent contributor a maintainer - a lot of open-source code and reliance on such code is based on trust. You could show some trust to active developers by making them maintainers. If it makes you feel better, you could ask them to sign some document so you could have some more confidence that they won't do anything malicious.

5. Support your community - promote unofficial libraries on your site. Here's how Airtable does it: https://airtable.com/developers/web/api/introduction
     Here are some Ruby projects that could be good candidates for adding to your website:
     - https://github.com/baoagency/polaris_view_components
     - https://github.com/travishaynes/shopify-mock/tree/httpmock
     - https://github.com/kirillplatonov/shopify-hotwire-sample
     - https://github.com/forsbergplustwo/partner-metrics-for-shopify
     - https://github.com/davefreiman/shopify-sample-data
     - https://github.com/kevinhughes27/shopify-tax-receipts
     - https://github.com/kirillplatonov/shopify_graphql
     - https://github.com/igrigorik/shopify-core-web-vitals
     - https://github.com/ScreenStaring/recharge-api
     - https://github.com/ScreenStaring/shopify_api_retry
     - https://github.com/lucidnz/lib-ruby-lucid-shopify
     - https://github.com/colinsoleim/shopify-metafieldsync
     - https://github.com/bravetheskies/polaris-html
     - https://github.com/ripenecommerce/shopify-ruby-sha256
     - https://github.com/remy727/rails-shopify-app-starter-kit
     - https://github.com/pcantarelli/app-ruby
     - https://github.com/jhnwr/shopify-scripts

6. Fix your release/changes workflow - a lot has been done in this area, but there's a lot to be desired. Where's the UPGRADING guide? Can you please add: "Do not release any breaking changes without upgrade steps" to your release workflow? Related issue from 4 years ago: Would be good to have MIGRATION_GUIDE.md or similar Would be good to have MIGRATION_GUIDE.md or similar shopify_app#774

7. Ask for requests for features - instead of guessing what your users (us, developers) need/want to see as new functionality, you could...just ask.

EDIT: 8. Remove stale bot.

Apologies if it sounds like I'm a bit frustrated, that's because I am. I've got several Shopify projects that I handle by myself and it's a pain to keep all of the projects up to date. Not to mention that every time I start a new Shopify-related project, the workflow and generated project file structure is completely different.

Adding in CC people whose issues I've mentioned. I'd love to get more input what can be done on getting ShopifyApp and Shopify API projects the much needed love they deserve.

@nelsonwittwer @matthewlein @guillelopezgines @Fladdermuz @jacksonSingleton @benichu @acflint @fmichaut-diff @ClaytonPassmore @garethson @nickmealey @flavio-b @jagthedrummer @marclennox @panckreous @srgoldman @LovroM @raymondweidner @raphaelfeitoza @amoughnie @andyw8

I keep seeing on this and other Shopify projects (Dawn theme, theme-check, CLI, etc.) the lack of good leadership and oversight. Seems like Shopify employees add features the way they want and no one is looking at the big picture. Stuff breaks, no one is talking to community, but the KPIs are met, because the tool X is now unified with CLI v3 or the gem now supports session auth, but we forgot the docs, sorry.

Please can someone from Shopify reply what are your plans with these projects? Do you have a roadmap? How many developer resources are dedicated to these projects? What is the plan on addressing these above-mentioned issues?
Don't leave me at "seen", I put more than 1 hour creating this issue. /bracing for stale bot 😄

[acflint]

Amen. Thank you for taking the time to write this up. I've basically stopped taking on new Shopify projects because things are such a mess. I would love to return to the ecosystem though, so I'm in full support of these requests. We won't let stale bot get its hands on this one.

[flavio-b]

All good suggestions! Also, I wonder if Shopify uses the same gems for their own apps and internal projects.

Maybe there's a slight disconnect here. I once reported what seemed like a major problem with the old GraphQL, and someone from Shopify wrote they used a different client internally, or at least had a different way to set it up, which meant they never caught the issue.

One of the best things about the Rails ecosystem is knowing you're using the very same tool that some pretty large companies rely on, so breaking changes are not taken lightly and important bugs are addressed quickly. I'm not sure if Shopify is using this gem in particular - correct me if I'm wrong.

I personally much prefer the new API library and the direction they chose, but leaving those issues without response impacts the developer experience for sure.

Would be great to see more resources dedicated to these gems, and make the development of the dev tooling a bit more even. New features get launched all the time, but some parts are left behind. I'm sure the developers involved in the tooling are doing their best, but the leadership is perhaps not focused on these issues enough and not providing the time or financial resources.

[vfonic]

I personally much prefer the new API library and the direction they chose, but leaving those issues without response impacts the developer experience for sure.

That's great to hear!

What's missing is UPGRADING.md / MIGRATION.md guide.

I'm sure the developers involved in the tooling are doing their best, but the leadership is perhaps not focused on these issues enough and not providing the time or financial resources.

I feel like this has been an issue at Shopify since forever.

[DannyBen]

I totally agree and am quite frustrated.

We have a long standing Shopify shop with a custom external Admin app using the API. I have spent hours now trying to figure out how do I authenticate it using access token and using the gem.

As I see it, I have no choice but to rewrite my app using standard API calls instead of relying on this gem with the documentation hell.

[nickmealey]

This needed to be said. We've been tripped up by this gem many times throughout the years. Building a Shopify app in some respect has gotten more difficult, especially when an app is embedded. It would be great to see more dedication toward keeping it well maintained.

[skillmatic-co]

Thank you for taking the time to write this. I 100% agree and I'm also frustrated. I have one Shopify client and even that's pushing my limits. I'm definitely not keen on taking on any other ones. Building apps here is a giant PITA. As I just mentioned in another issue, I'm still stuck at <V9 because the move from 9-10 literally broke everything in my app and it would need an entire rewrite to upgrade.

Are there any forks of a pre-v10 gem with improvements? Or has anyone built their own Shopify REST API wrapper that they are willing to share?

Getting feedback from Shopify here is pretty much nil it seems, I'll be curious to see if they respond to this - highly doubtful.

Everything you stated though is true - docs suck, upgrading sucks, changelogs suck. At times I think it's just a few greaseballs in a garage somewhere sadistically making breaking changes every month.

[nelsonwittwer]

We really appreciate the thoughtful comprehensive feedback on what the ruby community expects. Feedback is a gift and we treat this as such.

I've made some progress and will continue to iterate on the low hanging documentation improvements that you've pointed out. Stay tuned for more updates on how we can improve here; we are working on it!

[panckreous]

Hey @nelsonwittwer - thank you so much for responding to us.

In case it makes for an easier sell, here's the Rails commit that removed their auto-close bot.

rails/rails@acf4816

The commit message itself, penned by the venerable rafaelfranca:

While the idea of cleaning up the the PRs list by nudging reviewers with the stale message and closing PRs that didn't got a review in time cloud work for the maintainers, in practice it discourages contributors to submit contributions.

Keeping PRs open and not providing feedback also doesn't help with contributors motivation, so while I'm disabling this feature of the bot we still need to come up with a process that will help us to keep the number of PRs in check, but celebrate the work contributors already did instead of ignoring it, or dismissing in the form of a "stale" alerts, and automatically closing PRs.`

And a huge thank you to @vfonic for this masterpiece. Any chance you'd be willing to finish the Song of Ice and Fire novels next, since GRRM cant?

[vfonic]

Oh geez @panckreous, I almost believed you when you said "masterpiece", until you asked me to finish George Martin's novels. 😅

@nelsonwittwer I cannot hide the disappointment with the response from Shopify/you. After 3 weeks of silence, the reply you sent (as a Shopify representative) is: "thank you for your feedback, we're working on it". You could have a feedback answering machine compose that reply. I'm sorry for the sarcasm and if that sounds too harsh, but it's a reality. Zero information except "we merged some minor documentation PRs that have been opened for months".

So, it's been 6 weeks since I initially opened this issue and 3 weeks since you replied. When can we get some concrete information? What's Shopify's plan with Ruby? What's Shopify's plan with ShopifyApp and ShopifyAPI gems?

Was there nothing of value in my initial issue message? I've proposed several suggestions and pointed out several low-hanging fruit and I see that two documentation PRs were merged. So, ok, we've made some progress. Thank you!
...but at this rate, we'll see new version of ShopifyAPI gem (with complete rewrite of the gem and many breaking changes), much sooner than we'll see the actual documentation being updated...

How do we ensure that these issues don't happen again?

1. Can we have UPGRADING.md?
2. Can we have a mention of "new version cannot be published until the docs are updated!!!!111oneone" somewhere, maybe in RELEASING.md?
3. Can we have this piece of code (offline session) somewhere:

session = ShopifyAPI::Auth::Session.new(shop: shop.shopify_domain, access_token: shop.shopify_token)
ShopifyAPI::Context.activate_session(session)

4. Can we have stalebot removed, please? it's really tiring to take your time to describe an issue in great detail and then have it, first not answered by anyone, and second, automatically closed by stalebot as an insult to injury. Here's a good place to start answering stale issues: Why Shopify is not actively taking more care of these issues? #986
5. How about a roadmap?
6. Any plans of creating a platform for discussion? We could have Shopify Ruby developers community discuss about various topics: roadmap, feature requests, feedback, upcoming functionality, q&a (support), etc. But of course, it would be great to have someone from Shopify actively participating and moderating the discussions.

I know I've been rather negative in my comments, and I'm sorry. I really want to make something positive come out of this. I have huge respect for Shopify engineers and I know this is a lot of work. ❤️ I'm not saying anyone is slacking, I'm saying we need more resources.
That being said, I'll refrain from further commenting if not being directly asked.

[panckreous]

Oh geez @panckreous, I almost believed you when you said "masterpiece", until you asked me to finish George Martin's novels. 😅

oh no @vfonic that was an honest compliment. which, as often in real life too, I seemingly ruined via my terrible sense of humor. youve seen how many of us have tried over and over to no avail. you put the work and time in that finally got at least a response from one of the devs. I didnt think it was possible. I figured if you could accomplish that where no one else could...

I feel so bad now!

[nelsonwittwer]

You are right @vfonic , we owed you a better update(s) than what we have provided. Your tone and feedback is completely justified. Thank you again for you candor and your patience. This discussion has spurred many discussions on priorities internally and we couldn't have done so without your effort and feedback. The voice of the community has given us so much help in arguing our case for our root problem as you pointed out -- we need more resources. In all sincerity, I thank you.

The short update is this: I hope to share a vetted roadmap and change of philosophy/systems by the end of this week after we get final approvals on our internal changes we need to make to better serve the community.

I really wish I could have closed this loop sooner before you had respond; sadly getting approvals for more resources, etc take time.

[nelsonwittwer]

Based on feedback from the Ruby community and within Shopify, we will be making some changes to better encourage participation from external developers in how we move forward with our Ruby open sources packages. We have revisited our core open source principles and have a roadmap that we are excited to share and work on.

Build in public

Shopify will be adopting the mantra of building in public with all of you. This will include:

1. Sharing roadmap publicly to validate our direction with 1st and 3rd party Shopify developers.
2. Publishing a public specification for API client libraries, framework middleware (shopify_app gem), app templates, and REST resources. This specification will be used for all languages and frameworks to ensure a uniform scope and architecture.
3. Updating our contributors doc outlining our roadmap rituals and means of community interaction.

Roadmap

Moving forward our roadmap will be defined in the ROADMAP.md file in each of our repos. Our proposed overall Ruby roadmap include the following priorities over the next 6 weeks:

1. Update documentation for a) those getting started with using and/or contributing to the Ruby stack b) session management/usage and c) provide an upgrade doc, especially for those upgrading from v9.
2. Open sourcing the OpenAPI schemas and logic that generate the REST resources.
3. Extracting REST resources from the API client into a standalone package. These resources seem to be the root of many complaints with Sorbet, error handling, ActiveModel, etc. They also represent the biggest surface area for breaking changes. By extracting them into standalone libraries we can limit the number of breaking changes to the foundational API library as well as give a means for further modular customization for the community.
4. Update our release procedures to include a release candidate methodology for major releases. This will allow us to test our changes and documentation with our early adopters before introducing breaking changes.

We’d like to extend our thanks to the community for your patience and engagement. We recognize that we can’t do this alone and deeply value your feedback to build the best possible experience that meets the expectations of our developer community. We look forward to your contributions and suggestions!

[panckreous]

@nelsonwittwer can you please disable the stale-bot? #1103 has not been addressed and just got hit

[nelsonwittwer]

We have been brainstorming the stale bot @panckreous, but we can't commit to anything yet. The root problem of issue management is something we definitely still need to address. We plan on exploring that as we update CONTRIBUTING.md doc as part of our roadmap within the next 6 weeks.

Changes to issue management will be proposed in a PR. We look forward to discussing alternatives then :)