I preface
1.1. background
Use zookeeper to switch between active and standby services. Since the ticket of kerberos has a time cycle, in order to avoid uncontrollable impact of kerberos authentication on business, start verification for kerberos long connection
1.2. concept
Kerberos ticket has two life cycles, ticket timelife and renewable lifetime.
- When the ticket lifetime ends, the ticket will no longer be available.
- If renewable lifetime > ticket lifetime, the bill can be renewed throughout its life cycle until it reaches the upper limit of the renewable cycle.
- When the time reaches renewable lifetime, the ticket lifetime cannot be renewed after it ends. An error KDC can't fully requested option while renewing credentials will be reported during renewal. After that, you need to re apply for a new ticket.
- In terms of security, the advantage of renewable tickets over tickets with a longer life cycle is that KDC can refuse renewal requests (for example, if it is found that the account is damaged and the renewable tickets may be in the hands of attackers).
- The renewable cycle has nothing to do with keytabs. If you don't modify the relationship between key and principal, keytabs won't care.
For example:
- ticket_lifetime = 1d
- renew_lifetime = 7d
- The ticket can be renewed within 24 hours after login, and it will not be renewed until 7 days after the first login.
- If it is not renewed within 24h, it will not be renewed.
- After a renewal of the ticket, the ticket_lifetime will be restored to 24h.
II Environmental constraints
2.1. Environment version information
- Operating system: Centos 7
- kerberos version: Kerberos V5
- jdk version: 1.8
- Verify long connection service: zookeeper 4.4.8
2.2. krb5.conf configuration
[root@master01 ~]# more /etc/krb5.conf
# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
# The time limit for voucher effectiveness is set to 1 day.
ticket_lifetime = 24h
# The maximum time limit for the voucher to be extended is generally 7 days. After the certificate expires, subsequent access to the secure authentication service will fail.
renew_lifetime = 7d
# If this parameter is set to true, the ticket can be forwarded, which means that if the user with TGT logs in to the remote system, the KDC can issue a new TGT without requiring the user to authenticate again.
forwardable = true
rdns = false
pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
# Configure default realm
default_realm = EXAMPLE.COM
# default_ccache_name must comment out!!! Otherwise, the hadoop instruction will report an error:
# default_ccache_name = KEYRING:persistent:%{uid}
[realms]
# Domain specific information, such as the location of the Kerberos server for the domain. There may be several, one for each domain.
# You can specify a port for the KDC and the management server. If not configured,
# Then the KDC uses port 88 and the management server uses 749.
EXAMPLE.COM = {
kdc = master01:88
admin_server = master01:789
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
2.3. Test voucher list
- ZK service account
kadmin.local -q "addprinc -randkey zookeeper/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/zookeeper.keytab zookeeper/master01@EXAMPLE.COM " kadmin.local -q "addprinc -randkey zkcli/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/zkcli.keytab zkcli/master01@EXAMPLE.COM "
- Test account
kadmin.local -q "addprinc -randkey test01/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/test01.keytab test01/master01@EXAMPLE.COM " kadmin.local -q "addprinc -randkey test02/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/test02.keytab test02/master01@EXAMPLE.COM " kadmin.local -q "addprinc -randkey test03/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/test03.keytab test03/master01@EXAMPLE.COM " kadmin.local -q "addprinc -randkey test04/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/test04.keytab test04/master01@EXAMPLE.COM " kadmin.local -q "addprinc -randkey test05/master01@EXAMPLE.COM " kadmin.local -q "xst -k /opt/keytab/test05.keytab test05/master01@EXAMPLE.COM "
- Synchronize keytab files
scp /opt/keytab/* sysadmin@mac:/opt/keytab/
2.3. kerberos service operation instructions
- Start KDC server
# Enable krb5kdc systemctl enable krb5kdc # restart systemctl restart krb5kdc # start-up systemctl start krb5kdc # stop it systemctl stop krb5kdc #View status systemctl status krb5kdc #Set to boot systemctl enable krb5kdc.service
- Start kadmin service
# Enable kadmin systemctl enable kadmin # restart systemctl restart kadmin # start-up systemctl start kadmin # stop it systemctl stop kadmin #View status systemctl status kadmin #Set to boot systemctl enable kadmin.service
2.3. Krb5LoginModule parameter setting
Parameter reference:
https://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
III Results summary
For code solutions, refer to Zookeeper's code org apache. Zookeeper. Login .
The createconfiguratorframeworkclient here returns the CuratorFramework client. In fact, you can use the anonymous mode to directly access / read / write and start the zookeeper of kerberos Kerberos authentication is not required Therefore, you can refer to the method of createconfiguratorframeworkclient to establish the client. If it is a long connection, pay attention to starting the refresh thread kerberos clientproxy#startautorenew of the ticket
3.1.1 test code
package com.jaas;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.api.ACLProvider;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooDefs.Perms;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Id;
import org.apache.zookeeper.data.Stat;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import javax.security.auth.kerberos.KerberosTicket;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* After opening acl, verify zk connection is normal
*
* @author sysadmin
*/
public class ZookeeperAclClient {
private static final Logger logger = LoggerFactory.getLogger(ZookeeperAclClient.class);
public static void main(String[] args) throws Exception {
// Set krb5 Conf file address
System.setProperty("java.security.krb5.conf", KerberosOptions.TEST_KRB5_CONF_FILE);
// System.setProperty("sun.security.krb5.debug", "true");
String zkUserPwd = "admin:azadmin";
ACLProvider aclProvider = new ACLProvider() {
private List<ACL> acl;
@Override
public List<ACL> getDefaultAcl() {
if (acl == null) {
ArrayList<ACL> acl = ZooDefs.Ids.CREATOR_ALL_ACL;
acl.clear();
acl.add(new ACL(Perms.ALL, new Id("auth", zkUserPwd)));
this.acl = acl;
}
return acl;
}
@Override
public List<ACL> getAclForPath(String path) {
return acl;
}
};
String scheme = "digest";
byte[] auth = zkUserPwd.getBytes();
// Build curatorframeworkfactory Builder
// Set parameters as required
CuratorFrameworkFactory.Builder builder = CuratorFrameworkFactory.builder().connectString(KerberosOptions.DEFAULT_ZOOKEEPER_QUORUM)
.aclProvider(aclProvider)
.authorization(scheme, auth)
.retryPolicy(new ExponentialBackoffRetry(KerberosOptions.DEFAULT_ZOOKEEPER_RETRY_BASE_SLEEP,
KerberosOptions.DEFAULT_ZOOKEEPER_RETRY_MAXTIME, KerberosOptions.DEFAULT_ZOOKEEPER_RETRY_MAX_SLEEP))
.sessionTimeoutMs(KerberosOptions.DEFAULT_ZOOKEEPER_SESSION_TIMEOUT).connectionTimeoutMs(KerberosOptions.DEFAULT_ZOOKEEPER_CONNECTION_TIMEOUT);
KerberosClientProxy kerberosClientProxy = new KerberosClientProxy(KerberosOptions.TEST_PRINCIPAL, KerberosOptions.TEST_KEYTAB_FILE);
kerberosClientProxy.startAutoRenew();
CuratorFramework curatorClient = kerberosClientProxy.createCuratorFrameworkClient(builder);
long startTime = System.currentTimeMillis();
while (true) {
if (null != curatorClient.checkExists().forPath(KerberosOptions.TEST_ZK_DATA_PATH)) {
curatorClient.delete().forPath(KerberosOptions.TEST_ZK_DATA_PATH);
}
// The following is the test verification code
curatorClient.create().forPath(KerberosOptions.TEST_ZK_DATA_PATH, "hello world".getBytes());
byte[] bs = curatorClient.getData().forPath(KerberosOptions.TEST_ZK_DATA_PATH);
System.out.println("New node,data by: " + new String(bs));
curatorClient.setData().forPath(KerberosOptions.TEST_ZK_DATA_PATH, "hello china".getBytes());
// Since the data is obtained in the background mode, the bs may be null at this time
byte[] bs2 = curatorClient.getData().watched().inBackground().forPath(KerberosOptions.TEST_ZK_DATA_PATH);
if (bs2 == null) {
bs2 = curatorClient.getData().forPath(KerberosOptions.TEST_ZK_DATA_PATH);
}
System.out.println("Newly modified node,data by: " + new String(bs2 != null ? bs2 : new byte[0]));
curatorClient.delete().forPath(KerberosOptions.TEST_ZK_DATA_PATH);
Stat stat = curatorClient.checkExists().forPath(KerberosOptions.TEST_ZK_DATA_PATH);
// Stat is a mapping of all properties of zonde. stat=null means that the node does not exist!
System.out.println("Status after deleting a node: " + stat);
Set<KerberosTicket> tickets = kerberosClientProxy.getSubject().getPrivateCredentials(KerberosTicket.class);
for (KerberosTicket ticket : tickets) {
System.out.println(String.format("client survival time : [ %s ] s , AuthTime : [ %s ] , StartTime : [ %s ] , EndTime : [ %s ] ... ",
(System.currentTimeMillis() - startTime) / 1000, ticket.getAuthTime(), ticket.getStartTime(), ticket.getEndTime()));
}
kerberosClientProxy.reLogin();
TimeUnit.SECONDS.sleep(60);
}
}
}
3.2. Tool class code
package com.jaas;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import java.security.Principal;
import java.security.PrivilegedAction;
import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ThreadLocalRandom;
import javax.security.auth.Subject;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.kerberos.KerberosPrincipal;
import javax.security.auth.kerberos.KerberosTicket;
import javax.security.auth.login.AppConfigurationEntry;
import javax.security.auth.login.Configuration;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* @author sysadmin
*/
public class KerberosClientProxy {
private static final Logger logger = LoggerFactory.getLogger(KerberosClientProxy.class);
private static final String KERBEROS_LOGIN_CONTEXT_NAME = "Krb5Login";
// LoginThread will sleep until 80% of time from last refresh to
// ticket's expiry has been reached, at which time it will wake
// and try to renew the ticket.
private static final float TICKET_RENEW_WINDOW = 0.80f;
/**
* Percentage of random jitter added to the renewal time
*/
private static final float TICKET_RENEW_JITTER = 0.05f;
// Regardless of TICKET_RENEW_WINDOW setting above and the ticket expiry time,
// thread will not sleep between refresh attempts any less than 1 minute (60*1000 milliseconds = 1 minute).
// Change the '1' to e.g. 5, to change this to 5 minutes.
private static final long MIN_TIME_BEFORE_RELOGIN = 1 * 60 * 1000L;
public CallbackHandler callbackHandler;
private Subject subject = null;
private Thread autoRenewThread = null;
private LoginContext login = null;
private String principal = null;
private String keytabFileFullPath = null;
// Initialize 'lastLogin' to do a login at first time
private long lastLogin = currentElapsedTime() - MIN_TIME_BEFORE_RELOGIN;
KerberosClientProxy(String principal, String keytabFileFullPath) throws LoginException {
this(principal, keytabFileFullPath, null);
}
KerberosClientProxy(String principal, String keytabFileFullPath, CallbackHandler callbackHandler) throws LoginException {
this.principal = principal;
this.keytabFileFullPath = keytabFileFullPath;
this.callbackHandler = callbackHandler;
login = login();
subject = login.getSubject();
autoRenewThread = new Thread(() -> {
logger.info("TGT refresh thread started.");
while (true) { // renewal thread's main loop. if it exits from here, thread will exit.
KerberosTicket tgt = getTGT();
long now = currentWallTime();
long nextRefresh;
Date nextRefreshDate;
if (tgt == null) {
nextRefresh = now + MIN_TIME_BEFORE_RELOGIN;
nextRefreshDate = new Date(nextRefresh);
logger.warn("No TGT found: will try again at {}", nextRefreshDate);
} else {
nextRefresh = getRefreshTime(tgt);
long expiry = tgt.getEndTime().getTime();
Date expiryDate = new Date(expiry);
// determine how long to sleep from looking at ticket's expiry.
// We should not allow the ticket to expire, but we should take into consideration
// MIN_TIME_BEFORE_RELOGIN. Will not sleep less than MIN_TIME_BEFORE_RELOGIN, unless doing so
// would cause ticket expiration.
if ((nextRefresh > expiry) || ((now + MIN_TIME_BEFORE_RELOGIN) > expiry)) {
// expiry is before next scheduled refresh).
nextRefresh = now;
} else {
if (nextRefresh < (now + MIN_TIME_BEFORE_RELOGIN)) {
// next scheduled refresh is sooner than (now + MIN_TIME_BEFORE_LOGIN).
Date until = new Date(nextRefresh);
Date newuntil = new Date(now + MIN_TIME_BEFORE_RELOGIN);
logger.warn(String.format("TGT refresh thread time adjusted from : {%s} to : {%s} since "
+ "the former is sooner than the minimum refresh interval ("
+ "{%s} seconds) from now.", until,
newuntil,
(MIN_TIME_BEFORE_RELOGIN / 1000)));
}
nextRefresh = Math.max(nextRefresh, now + MIN_TIME_BEFORE_RELOGIN);
}
nextRefreshDate = new Date(nextRefresh);
if (nextRefresh > expiry) {
logger.error(
"next refresh: {} is later than expiry {}."
+ " This may indicate a clock skew problem."
+ " Check that this host and the KDC's "
+ "hosts' clocks are in sync. Exiting refresh thread.",
nextRefreshDate,
expiryDate);
return;
}
}
if (now == nextRefresh) {
logger.info("refreshing now because expiry is before next scheduled refresh time.");
} else if (now < nextRefresh) {
Date until = new Date(nextRefresh);
logger.info("TGT refresh sleeping until: {}", until);
try {
Thread.sleep(nextRefresh - now);
} catch (InterruptedException ie) {
logger.warn("TGT renewal thread has been interrupted and will exit.");
break;
}
} else {
logger.error(
"nextRefresh:{} is in the past: exiting refresh thread. Check"
+ " clock sync between this host and KDC - (KDC's clock is likely ahead of this host)."
+ " Manual intervention will be required for this client to successfully authenticate."
+ " Exiting refresh thread.",
nextRefreshDate);
break;
}
try {
int retry = 1;
while (retry >= 0) {
try {
reLogin();
break;
} catch (LoginException le) {
if (retry > 0) {
--retry;
// sleep for 10 seconds.
try {
Thread.sleep(10 * 1000);
} catch (InterruptedException e) {
logger.error("Interrupted during login retry after LoginException:", le);
throw le;
}
} else {
logger.error("Could not refresh TGT for principal: {}.", principal, le);
}
}
}
} catch (LoginException le) {
logger.error("Failed to refresh TGT: refresh thread exiting now.", le);
break;
}
}
});
autoRenewThread.setDaemon(true);
}
private Configuration getConfiguration() {
return new Configuration() {
@Override
public AppConfigurationEntry[] getAppConfigurationEntry(String name) {
HashMap<String, Object> options = new HashMap<String, Object>(16) {
{
put("useTicketCache", "false");
put("renewTGT", "false");
put("useKeyTab", "true");
//Krb5 in GSS API needs to be refreshed so it does not throw the error
//Specified version of key is not available
put("refreshKrb5Config", "true");
put("storeKey", "true");
put("doNotPrompt", "true");
put("isInitiator", "true");
put("principal", principal);
put("keyTab", keytabFileFullPath);
}
};
return new AppConfigurationEntry[]{new AppConfigurationEntry("com.sun.security.auth.module.Krb5LoginModule",
AppConfigurationEntry.LoginModuleControlFlag.REQUIRED, options)};
}
};
}
private LoginContext login() throws LoginException {
Set<Principal> principalSet = new HashSet<>(1);
principalSet.add(new KerberosPrincipal(this.principal));
Subject sub = new Subject(false, principalSet, new HashSet<>(), new HashSet<>());
LoginContext login = new LoginContext(KERBEROS_LOGIN_CONTEXT_NAME, sub, callbackHandler, getConfiguration());
login.login();
logger.info("{} successfully logged in.", principal);
return login;
}
public void startAutoRenew() {
autoRenewThread.start();
}
public Subject getSubject() {
return subject;
}
public String getPrincipal() {
return principal;
}
public void shutdown() {
if ((autoRenewThread != null) && (autoRenewThread.isAlive())) {
autoRenewThread.interrupt();
try {
autoRenewThread.join();
} catch (InterruptedException e) {
logger.warn("error while waiting for Login thread to shutdown.", e);
}
}
}
// c.f. org.apache.hadoop.security.UserGroupInformation.
private long getRefreshTime(KerberosTicket tgt) {
long start = tgt.getStartTime().getTime();
long expires = tgt.getEndTime().getTime();
logger.info("TGT valid starting at: {}", tgt.getStartTime().toString());
logger.info("TGT expires: {}", tgt.getEndTime().toString());
long proposedRefresh = start + (long) ((expires - start)
* (TICKET_RENEW_WINDOW + (TICKET_RENEW_JITTER
* ThreadLocalRandom.current().nextDouble())));
if (proposedRefresh > expires) {
// proposedRefresh is too far in the future: it's after ticket expires: simply return now.
return currentWallTime();
} else {
return proposedRefresh;
}
}
private synchronized KerberosTicket getTGT() {
Set<KerberosTicket> tickets = subject.getPrivateCredentials(KerberosTicket.class);
for (KerberosTicket ticket : tickets) {
KerberosPrincipal client = ticket.getClient();
if (client.getName().equals(principal)) {
logger.debug("Client principal is \"{}\".", ticket.getClient().getName());
logger.debug("Server principal is \"{}\".", ticket.getServer().getName());
return ticket;
}
}
return null;
}
private boolean hasSufficientTimeElapsed() {
long now = currentElapsedTime();
if (now - getLastLogin() < MIN_TIME_BEFORE_RELOGIN) {
logger.warn("Not attempting to re-login since the last re-login was "
+ "attempted less than {} seconds before.",
(MIN_TIME_BEFORE_RELOGIN / 1000));
return false;
}
// register most recent relogin attempt
setLastLogin(now);
return true;
}
/**
* Get the time of the last login.
*
* @return the number of milliseconds since the beginning of time.
*/
private long getLastLogin() {
return lastLogin;
}
/**
* Set the last login time.
*
* @param time the number of milliseconds since the beginning of time
*/
private void setLastLogin(long time) {
lastLogin = time;
}
/**
* Returns login object
*
* @return login
*/
private LoginContext getLogin() {
return login;
}
/**
* Set the login object
*/
private void setLogin(LoginContext login) {
this.login = login;
}
/**
* Re-login a principal. This method assumes that {@link #KerberosClientProxy(String, String)} has happened already.
*
* @throws javax.security.auth.login.LoginException on a failure
*/
// c.f. HADOOP-6559
public synchronized void reLogin() throws LoginException {
LoginContext login = getLogin();
if (login == null) {
throw new LoginException("login must be done first");
}
if (!hasSufficientTimeElapsed()) {
return;
}
logger.debug("Initiating logout for {}", principal);
synchronized (KerberosClientProxy.class) {
//clear up the kerberos state. But the tokens are not cleared! As per
//the Java kerberos login module code, only the kerberos credentials
//are cleared
login.logout();
//login and also update the subject field of this instance to
//have the new credentials (pass it to the LoginContext constructor)
login = new LoginContext(KERBEROS_LOGIN_CONTEXT_NAME, getSubject(),callbackHandler,getConfiguration());
logger.debug("Initiating re-login for {}", principal);
login.login();
setLogin(login);
}
}
public synchronized CuratorFramework createCuratorFrameworkClient(CuratorFrameworkFactory.Builder builder) {
return Subject.doAs(getSubject(), (PrivilegedAction<CuratorFramework>) () -> {
try {
CuratorFramework client = builder.build();
client.start();
logger.info("Start blocking connection ZK .... ");
client.blockUntilConnected();
logger.info("Connection to ZK succeeded .... ");
return client;
} catch (Exception e) {
e.printStackTrace();
}
return null;
});
}
/**
* Returns time in milliseconds as does System.currentTimeMillis(),
* but uses elapsed time from an arbitrary epoch more like System.nanoTime().
* The difference is that if somebody changes the system clock,
* currentElapsedTime will change but nanoTime won't. On the other hand,
* all of ZK assumes that time is measured in milliseconds.
*
* @return The time in milliseconds from some arbitrary point in time.
*/
public long currentElapsedTime() {
return System.nanoTime() / 1000000;
}
/**
* Explicitly returns system dependent current wall time.
*
* @return Current time in msec.
*/
public long currentWallTime() {
return System.currentTimeMillis();
}
}