Disabled feature flag returns "true" from time to time

[tiliev]

I have an issue with a disabled feature flag that returns true from time to time.
In our Azure App config I have two flags with the same name:

• Flag1 with no label and value true
• Flag1 with environment prod and value false

Sometimes on environment prod I get value true for Flag1.

Our setup:

• We do not want to load all of the flags with no label. That's why we do options.Select("_").
• I'm using our feature flag manager inside of singleton class
• We work with feature flags in appsettings.Development.json when developing locally

Code:

Program.cs

webBuilder.ConfigureAppConfiguration(config =>
{
  var settings = config.Build();
  var azureAppConfig = settings.GetSection("AzureAppConfig").Get<AzureAppConfiguration>() ?? new();
  if (string.IsNullOrWhiteSpace(azureAppConfig.ConnectionString))
      return;
  
  config.AddAzureAppConfiguration(options =>
  {
      options.Select("_");
  
      options
          .Connect(azureAppConfig.ConnectionString)
          .UseFeatureFlags(ffOptions =>
          { 
              ffOptions.Select("Service*", LabelFilter.Null);
              ffOptions.Select("Service*", azureAppConfig.LabelFilter);
          });
  });
});

Startup.cs

public static IServiceCollection AddFeatureFlags(this IServiceCollection services, IConfiguration configuration)
{
    var azureAppConfig = configuration.GetSection("AzureAppConfig").Get<AzureAppConfiguration>() ?? new();
    var enableAzureAppConfig = !string.IsNullOrWhiteSpace(azureAppConfig.ConnectionString);

    if (enableAzureAppConfig)
        services.AddAzureAppConfiguration();
    else
        services.AddSingleton<IConfigurationRefresherProvider, NullConfigurationRefresherProvider>();

    services.AddFeatureManagement();
    
    return services;
}

Usage:

public class FeatureFlagsManager : IFeatureFlagsManager
{
    private readonly IFeatureManager featureManager;
    private readonly IEnumerable<IConfigurationRefresher> refreshers;

    public FeatureFlagsManager(IFeatureManager featureManager, IConfigurationRefresherProvider configurationRefresherProvider)
    {
        this.featureManager = featureManager;
        refreshers = configurationRefresherProvider.Refreshers;
    }

    public async Task<bool> IsEnabledAsync(string feature)
    {
        await RefreshConfiguration();

        return await featureManager.IsEnabledAsync(feature);
    }

    private async Task RefreshConfiguration()
    {
        await Task.WhenAll(refreshers.Select(r => r.RefreshAsync()));
    }
}

Please let us know if you see any issues with our setup.

[jimmyca15]

@tiliev Do you modify feature flags with null label while the application is running. And is this what causes the symptom? Or does this happen even if no changes are made to feature flags?

[tiliev]

@jimmyca15 in the beginning I had both flags (with and without the label) set to true. I can't be sure whether it has worked for 100% of the time though. While the application was running, I turned off the flag with a label and noticed that the flag returned true once (~24 h after the change).

Another question - is it advisable to use a configuration like ours - supporting default values (no labels) + environment values (with labels)? Or is it better to stick with labels only and do not use flags without labels? We are AAC for feature flags only.

[jimmyca15]

@tiliev I'm able to reproduce a similar issue by having a slightly different situation than you described. If I have the following flags, and loading feature flags with your setup:

• Flag1 with no label and value true
• Flag1 with environment prod and value false

The application will start with Flag1 as false. However, if Flag1 with no label is modified during the runtime of the application it will override whatever value Flag1 with label prod has. Even if that modification doesn't change the underlying value of the flag (true vs false).

This behavior may be unexpected since the original feature flag query is setup so that flags with the label prod override flags with no label.

Other than that, I'm unable to reproduce this issue. Could it be possible that you have a deployment running that idempotently updates Flag1 with no label at some interval?

[tiliev]

@jimmyca15 thanks for looking into this so quickly.

I think what you describe is a bug.

To your question - nope, we update feature flags via the UI only. Services have read-only permissions.

I suspected there might be some issues with the overriding part and I simplified our configuration:

config.AddAzureAppConfiguration(options =>
{
    options.Select("_");
  
    options
        .Connect(azureAppConfig.ConnectionString)
        .UseFeatureFlags(ffOptions =>
        {
            ffOptions.Select("Service*", azureAppConfig.LabelFilter);
        });
});

These are our settings in the UI - that's the same key:

So, today (on 9th) in the morning on environment production on its first call to AAC that flag returned false. All of the subsequent requests were with the correct value - true. So, even with a simplified configuration, it doesn't work correctly for 100% of the time.

I'll be happy if you could provide us with more information regarding the following:

1. Do you think our configuration is suitable for our use case? Would you recommend doing something else?
2. Could this line options.Select("_"); introduce some issues? I saw it in another thread as a recommendation from you when a user do not want to load all of the flags without label.
3. Is AAC expected to behave correctly in a distributed environment where we have many instances of the same application? I think you use memory cache and each instance has its own cache, but can you confirm that is indeed the case?

Thank you for your time.

[jimmyca15]

on environment production on its first call to AAC that flag returned false

Do you mean you had a single web app, and you evaluated the flag multiple times? Or do you mean you had multiple different applications and in one of them the flag was false?

With the symptoms you are describing in your simplified scenario, the following code should be able to reproduce the issue:

string appConfigurationEndpoint = "";
string featurePrefix = "";
string environment = ""; // used for feature flag label

var serviceCollection = new ServiceCollection();

var configurationBuilder = new ConfigurationBuilder();

configurationBuilder.AddAzureAppConfiguration(aacOptions =>
{
    aacOptions.Connect(new Uri(appConfigurationEndpoint), new VisualStudioCredential());

    aacOptions.Select("_");

    aacOptions.UseFeatureFlags(fo =>
    {
        fo.Select("H*", environment);
    });
});

IConfiguration configuration = configurationBuilder.Build();

serviceCollection.AddSingleton(configuration);

serviceCollection.AddFeatureManagement();

using ServiceProvider provider = serviceCollection.BuildServiceProvider();

IFeatureManager featureManager = provider.GetRequiredService<IFeatureManager>();

Console.WriteLine(await featureManager.IsEnabledAsync("Home"));

I don't see it happening in repeated iterations. Also, unable to find a code defect which could produce the behavior.

1. The configuration is suitable. The only problem is the behavior I mentioned above that can occur if you modify the feature flag with no label during the runtime of the application. In the current code, a refresh operation will cause that flag to override the flag with the label. That behavior should be addressed on our side. If you're not updating the feature flag that has no label it shouldn't be problematic.

2. That shouldn't cause any problems. That is the right approach to take.

3. Yes AAC should be fine in that case. Yes each application stores feature flag state in memory.

[tiliev]

Hello, our use case for the time being is one application on multiple nodes.

We reduced our configuration even more and we do not use feature flags without a label anymore:

config.AddAzureAppConfiguration(options =>
{
    options
        .Connect(azureAppConfig.ConnectionString)
        .UseFeatureFlags(ffOptions =>
        {
            ffOptions.Select("Service*", azureAppConfig.LabelFilter);
        });
});

This appears to be working 100% although we'll monitor the results for some more time.

Could you let us know when the defect with the flags with default values is fixed so we can test it on our side?

Thanks a lot for your help.

[zhenlan]

@tiliev I'm able to reproduce a similar issue by having a slightly different situation than you described.

@jimmyca15 is there an issue opened to track the issue you mentioned earlier? Is that the same issue @tiliev mentioned above?

[jimmyca15]

@zhenlan

No issue was ever filed. I filed it here: Azure/AppConfiguration-DotnetProvider#317

[zhenlan]

Cool. Thanks, @jimmyca15. I am closing this one since the issue discovered in this thread is tracked separately.