Skip to content

HDDS-2162. Make OM Generic related configuration support HA style config. #1511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 2, 2019
Merged

HDDS-2162. Make OM Generic related configuration support HA style config. #1511

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8a1593f
HDDS-2162. Make Kerberos related configuration support HA style config.
bharatviswa504 Sep 24, 2019
c4e7666
Few more changes required for HA/Non-HA
bharatviswa504 Sep 24, 2019
dff8c42
address review comments.
bharatviswa504 Sep 25, 2019
0d1d9b2
remove kerberos principal.
bharatviswa504 Sep 27, 2019
821332f
fix test
bharatviswa504 Oct 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -404,18 +404,14 @@ private static void addFilesToArchive(String source, File file,
/**
* If a OM conf is only set with key suffixed with OM Node ID, return the
* set value.
* @return null if base conf key is set, otherwise the value set for
* key suffixed with Node ID.
* @return if the value is set for key suffixed with OM Node ID, return the
* value, else return null.
*/
public static String getConfSuffixedWithOMNodeId(Configuration conf,
String confKey, String omServiceID, String omNodeId) {
String confValue = conf.getTrimmed(confKey);
if (StringUtils.isNotEmpty(confValue)) {
return null;
}
String suffixedConfKey = OmUtils.addKeySuffixes(
confKey, omServiceID, omNodeId);
confValue = conf.getTrimmed(suffixedConfKey);
String confValue = conf.getTrimmed(suffixedConfKey);
if (StringUtils.isNotEmpty(confValue)) {
return confValue;
}
Expand Down
9 changes: 6 additions & 3 deletions ...egration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,13 @@ public void testDefaultPortIfNotSpecified() throws Exception {
String omNode1Id = "omNode1";
String omNode2Id = "omNode2";
String omNodesKeyValue = omNode1Id + "," + omNode2Id;
conf.set(OMConfigKeys.OZONE_OM_NODES_KEY, omNodesKeyValue);
String serviceID = "service1";
conf.set(OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY, serviceID);
conf.set(OMConfigKeys.OZONE_OM_NODES_KEY + "." + serviceID,
omNodesKeyValue);

String omNode1RpcAddrKey = getOMAddrKeyWithSuffix(null, omNode1Id);
String omNode2RpcAddrKey = getOMAddrKeyWithSuffix(null, omNode2Id);
String omNode1RpcAddrKey = getOMAddrKeyWithSuffix(serviceID, omNode1Id);
String omNode2RpcAddrKey = getOMAddrKeyWithSuffix(serviceID, omNode2Id);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't need to verify that these strings are in the expected format here?

Copy link
Contributor Author

@bharatviswa504 bharatviswa504 Sep 26, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have an uber comment on this JIRA. Under Ozone, what we really need is 3 + 3 six kerberos Identites.

Why don't we just follow the standard Kerberos SPN names? Simple take one config key from the user, either the SPN or file name path the service kerberos identity.

Once you have this, we don't have to do any second guessing or munching of names with any other strings -- after all it is just a service on a host. Code is simpler, and the best part, it is simple enough for any one to understand. In other words, what evil are we trying to prevent here with all the service name munching ?

Not got your last part what is proposed.

This is done for Kerberos settings and also for other configs like OM DB DIrs Http/Https Address. Suppose the user wants to use different keytab file location/principal name it will also help in this situation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not got your last part what is proposed.

https://docs.microsoft.com/en-us/windows/win32/ad/name-formats-for-unique-spns

OzoneManager/host1.example.com/CN=hrdb,OU=mktg,DC=example,DC=com
OzoneManager/host2.example.com/CN=hrdb,OU=mktg,DC=example,DC=com
OzoneManager/host3.example.com/CN=hrdb,OU=mktg,DC=example,DC=com

This is all we need, is what I am trying to say.

Suppose the user wants to use different keytab file location/principal name it will also help in this situation.

Why would you want separate identities to communicate to the same service ? Can you give me an example of why this would be needed ? More over, why support that identity via naming tricks in Ozone instead of creating an new SPN in Kerberos Domain?

Copy link
Contributor Author

@bharatviswa504 bharatviswa504 Sep 27, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline, from my understanding this is being done as to share config across all OM's. And this PR is not changing any config loading code of OM HA, it just added Kerberos/DB config as described in Jira description.

Anu said we don't require the current way, and we shall continue the discussion later to see how we can do it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline for now removed Kerberos config.

conf.set(omNode1RpcAddrKey, "0.0.0.0");
conf.set(omNode2RpcAddrKey, "122.0.0.122");
Expand Down
230 changes: 28 additions & 202 deletions hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@
import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension;
import org.apache.hadoop.fs.CommonConfigurationKeys;
import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdds.HddsConfigKeys;
import org.apache.hadoop.hdds.HddsUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
Expand Down Expand Up @@ -74,9 +73,10 @@
import org.apache.hadoop.ipc.Server;
import org.apache.hadoop.ozone.OzoneAcl;
import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.OzoneIllegalArgumentException;
import org.apache.hadoop.ozone.OzoneSecurityUtil;
import org.apache.hadoop.ozone.om.ha.OMFailoverProxyProvider;
import org.apache.hadoop.ozone.om.ha.OMHANodeDetails;
import org.apache.hadoop.ozone.om.ha.OMNodeDetails;
import org.apache.hadoop.ozone.om.helpers.OmMultipartUploadList;
import org.apache.hadoop.ozone.om.helpers.S3SecretValue;
import org.apache.hadoop.ozone.om.protocol.OzoneManagerServerProtocol;
Expand Down Expand Up @@ -207,10 +207,6 @@
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_KERBEROS_PRINCIPAL_KEY;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_METRICS_SAVE_INTERVAL;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_METRICS_SAVE_INTERVAL_DEFAULT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_NODE_ID_KEY;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_RATIS_PORT_DEFAULT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_SERVICE_IDS_KEY;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_RATIS_PORT_KEY;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_USER_MAX_VOLUME;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_USER_MAX_VOLUME_DEFAULT;
import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_AUTH_METHOD;
Expand Down Expand Up @@ -310,12 +306,32 @@ private OzoneManager(OzoneConfiguration conf) throws IOException,
super(OzoneVersionInfo.OZONE_VERSION_INFO);
Preconditions.checkNotNull(conf);
configuration = conf;
// Load HA related configurations
OMHANodeDetails omhaNodeDetails =
OMHANodeDetails.loadOMHAConfig(configuration);

this.peerNodes = omhaNodeDetails.getPeerNodeDetails();
this.omNodeDetails = omhaNodeDetails.getLocalNodeDetails();

omStorage = new OMStorage(conf);
omId = omStorage.getOmId();

// In case of single OM Node Service there will be no OM Node ID
// specified, set it to value from om storage
if (this.omNodeDetails.getOMNodeId() == null) {
this.omNodeDetails =
OMHANodeDetails.getOMNodeDetails(conf, omNodeDetails.getOMServiceId(),
omStorage.getOmId(), omNodeDetails.getRpcAddress(),
omNodeDetails.getRatisPort());
}

loginOMUserIfSecurityEnabled(conf);

this.maxUserVolumeCount = conf.getInt(OZONE_OM_USER_MAX_VOLUME,
OZONE_OM_USER_MAX_VOLUME_DEFAULT);
Preconditions.checkArgument(this.maxUserVolumeCount > 0,
OZONE_OM_USER_MAX_VOLUME + " value should be greater than zero");
omStorage = new OMStorage(conf);
omId = omStorage.getOmId();

if (omStorage.getState() != StorageState.INITIALIZED) {
throw new OMException("OM not initialized.",
ResultCodes.OM_NOT_INITIALIZED);
Expand All @@ -342,8 +358,7 @@ private OzoneManager(OzoneConfiguration conf) throws IOException,
OMConfigKeys.OZONE_OM_RATIS_ENABLE_KEY,
OMConfigKeys.OZONE_OM_RATIS_ENABLE_DEFAULT);

// Load HA related configurations
loadOMHAConfigs(configuration);

InetSocketAddress omNodeRpcAddr = omNodeDetails.getRpcAddress();
omRpcAddressTxt = new Text(omNodeDetails.getRpcAddressString());

Expand Down Expand Up @@ -420,7 +435,7 @@ private OzoneManager(OzoneConfiguration conf) throws IOException,
OzoneManagerProtocolProtos.Type.values());

// Start Om Rpc Server.
omRpcServer = getRpcServer(conf);
omRpcServer = getRpcServer(configuration);
omRpcAddress = updateRPCListenAddress(configuration,
OZONE_OM_ADDRESS_KEY, omNodeRpcAddr, omRpcServer);

Expand Down Expand Up @@ -513,195 +528,6 @@ public boolean isGrpcBlockTokenEnabled() {
return grpcBlockTokenEnabled;
}

/**
* Inspects and loads OM node configurations.
*
* If {@link OMConfigKeys#OZONE_OM_SERVICE_IDS_KEY} is configured with
* multiple ids and/ or if {@link OMConfigKeys#OZONE_OM_NODE_ID_KEY} is not
* specifically configured , this method determines the omServiceId
* and omNodeId by matching the node's address with the configured
* addresses. When a match is found, it sets the omServicId and omNodeId from
* the corresponding configuration key. This method also finds the OM peers
* nodes belonging to the same OM service.
*
* @param conf
*/
private void loadOMHAConfigs(Configuration conf) {
InetSocketAddress localRpcAddress = null;
String localOMServiceId = null;
String localOMNodeId = null;
int localRatisPort = 0;
Collection<String> omServiceIds = conf.getTrimmedStringCollection(
OZONE_OM_SERVICE_IDS_KEY);

String knownOMNodeId = conf.get(OZONE_OM_NODE_ID_KEY);
int found = 0;
boolean isOMAddressSet = false;

for (String serviceId : OmUtils.emptyAsSingletonNull(omServiceIds)) {
Collection<String> omNodeIds = OmUtils.getOMNodeIds(conf, serviceId);

List<OMNodeDetails> peerNodesList = new ArrayList<>();
boolean isPeer = false;
for (String nodeId : OmUtils.emptyAsSingletonNull(omNodeIds)) {
if (knownOMNodeId != null && !knownOMNodeId.equals(nodeId)) {
isPeer = true;
} else {
isPeer = false;
}
String rpcAddrKey = OmUtils.addKeySuffixes(OZONE_OM_ADDRESS_KEY,
serviceId, nodeId);
String rpcAddrStr = OmUtils.getOmRpcAddress(conf, rpcAddrKey);
if (rpcAddrStr == null) {
continue;
}

// If OM address is set for any node id, we will not fallback to the
// default
isOMAddressSet = true;

String ratisPortKey = OmUtils.addKeySuffixes(OZONE_OM_RATIS_PORT_KEY,
serviceId, nodeId);
int ratisPort = conf.getInt(ratisPortKey, OZONE_OM_RATIS_PORT_DEFAULT);

InetSocketAddress addr = null;
try {
addr = NetUtils.createSocketAddr(rpcAddrStr);
} catch (Exception e) {
LOG.warn("Exception in creating socket address " + addr, e);
continue;
}
if (!addr.isUnresolved()) {
if (!isPeer && OmUtils.isAddressLocal(addr)) {
localRpcAddress = addr;
localOMServiceId = serviceId;
localOMNodeId = nodeId;
localRatisPort = ratisPort;
found++;
} else {
// This OMNode belongs to same OM service as the current OMNode.
// Add it to peerNodes list.
String httpAddr = OmUtils.getHttpAddressForOMPeerNode(conf,
serviceId, nodeId, addr.getHostName());
String httpsAddr = OmUtils.getHttpsAddressForOMPeerNode(conf,
serviceId, nodeId, addr.getHostName());
OMNodeDetails peerNodeInfo = new OMNodeDetails.Builder()
.setOMServiceId(serviceId)
.setOMNodeId(nodeId)
.setRpcAddress(addr)
.setRatisPort(ratisPort)
.setHttpAddress(httpAddr)
.setHttpsAddress(httpsAddr)
.build();
peerNodesList.add(peerNodeInfo);
}
}
}
if (found == 1) {
LOG.debug("Found one matching OM address with service ID: {} and node" +
" ID: {}", localOMServiceId, localOMNodeId);

setOMNodeDetails(localOMServiceId, localOMNodeId, localRpcAddress,
localRatisPort);

this.peerNodes = peerNodesList;

LOG.info("Found matching OM address with OMServiceId: {}, " +
"OMNodeId: {}, RPC Address: {} and Ratis port: {}",
localOMServiceId, localOMNodeId,
NetUtils.getHostPortString(localRpcAddress), localRatisPort);
return;
} else if (found > 1) {
String msg = "Configuration has multiple " + OZONE_OM_ADDRESS_KEY +
" addresses that match local node's address. Please configure the" +
" system with " + OZONE_OM_SERVICE_IDS_KEY + " and " +
OZONE_OM_ADDRESS_KEY;
throw new OzoneIllegalArgumentException(msg);
}
}

if (!isOMAddressSet) {
// No OM address is set. Fallback to default
InetSocketAddress omAddress = OmUtils.getOmAddress(conf);
int ratisPort = conf.getInt(OZONE_OM_RATIS_PORT_KEY,
OZONE_OM_RATIS_PORT_DEFAULT);

LOG.info("Configuration either no {} set. Falling back to the default " +
"OM address {}", OZONE_OM_ADDRESS_KEY, omAddress);

setOMNodeDetails(null, null, omAddress, ratisPort);

} else {
String msg = "Configuration has no " + OZONE_OM_ADDRESS_KEY + " " +
"address that matches local node's address. Please configure the " +
"system with " + OZONE_OM_ADDRESS_KEY;
LOG.info(msg);
throw new OzoneIllegalArgumentException(msg);
}
}

/**
* Builds and sets OMNodeDetails object.
*/
private void setOMNodeDetails(String serviceId, String nodeId,
InetSocketAddress rpcAddress, int ratisPort) {

if (serviceId == null) {
// If no serviceId is set, take the default serviceID om-service
serviceId = OzoneConsts.OM_SERVICE_ID_DEFAULT;
LOG.info("OM Service ID is not set. Setting it to the default ID: {}",
serviceId);
}
if (nodeId == null) {
// If no nodeId is set, take the omId from omStorage as the nodeID
nodeId = omId;
LOG.info("OM Node ID is not set. Setting it to the OmStorage's " +
"OmID: {}", nodeId);
}

this.omNodeDetails = new OMNodeDetails.Builder()
.setOMServiceId(serviceId)
.setOMNodeId(nodeId)
.setRpcAddress(rpcAddress)
.setRatisPort(ratisPort)
.build();

// Set this nodes OZONE_OM_ADDRESS_KEY to the discovered address.
configuration.set(OZONE_OM_ADDRESS_KEY,
NetUtils.getHostPortString(rpcAddress));

// Get and set Http(s) address of local node. If base config keys are
// not set, check for keys suffixed with OM serivce ID and node ID.
setOMNodeSpecificConfigs(serviceId, nodeId);
}

/**
* Check if any of the following configuration keys have been set using OM
* Node ID suffixed to the key. If yes, then set the base key with the
* configured valued.
* 1. {@link OMConfigKeys#OZONE_OM_HTTP_ADDRESS_KEY}
* 2. {@link OMConfigKeys#OZONE_OM_HTTPS_ADDRESS_KEY}
* 3. {@link OMConfigKeys#OZONE_OM_HTTP_BIND_HOST_KEY}
* 4. {@link OMConfigKeys#OZONE_OM_HTTPS_BIND_HOST_KEY}
*/
private void setOMNodeSpecificConfigs(String omServiceId, String omNodeId) {
String[] confKeys = new String[] {
OMConfigKeys.OZONE_OM_HTTP_ADDRESS_KEY,
OMConfigKeys.OZONE_OM_HTTPS_ADDRESS_KEY,
OMConfigKeys.OZONE_OM_HTTP_BIND_HOST_KEY,
OMConfigKeys.OZONE_OM_HTTPS_BIND_HOST_KEY};

for (String confKey : confKeys) {
String confValue = OmUtils.getConfSuffixedWithOMNodeId(
configuration, confKey, omServiceId, omNodeId);
if (confValue != null) {
LOG.info("Setting configuration key {} with value of key {}: {}",
confKey, OmUtils.addKeySuffixes(confKey, omNodeId), confValue);
configuration.set(confKey, confValue);
}
}
}

private KeyProviderCryptoExtension createKeyProviderExt(
OzoneConfiguration conf) throws IOException {
KeyProvider keyProvider = KMSUtil.createKeyProvider(conf,
Expand Down Expand Up @@ -1024,7 +850,6 @@ private static boolean isOzoneSecurityEnabled() {
*/
public static OzoneManager createOm(OzoneConfiguration conf)
throws IOException, AuthenticationException {
loginOMUserIfSecurityEnabled(conf);
return new OzoneManager(conf);
}

Expand Down Expand Up @@ -1053,6 +878,7 @@ private static void loginOMUserIfSecurityEnabled(OzoneConfiguration conf)
@VisibleForTesting
public static boolean omInit(OzoneConfiguration conf) throws IOException,
AuthenticationException {
OMHANodeDetails.loadOMHAConfig(conf);
loginOMUserIfSecurityEnabled(conf);
OMStorage omStorage = new OMStorage(conf);
StorageState state = omStorage.getState();
Expand Down Expand Up @@ -1361,7 +1187,7 @@ private RPC.Server getRpcServer(OzoneConfiguration conf) throws IOException {
return omRpcServer;
}

InetSocketAddress omNodeRpcAddr = OmUtils.getOmAddress(configuration);
InetSocketAddress omNodeRpcAddr = OmUtils.getOmAddress(conf);

final int handlerCount = conf.getInt(OZONE_OM_HANDLER_COUNT_KEY,
OZONE_OM_HANDLER_COUNT_DEFAULT);
Expand Down
Loading