SpringBoot线上服务假死，CPU内存正常，排查过程

问题排查

常规排查

top -H -p pid
或
ps -mp pid -o THREAD,tid,time

printf “%x\n” nid

jstack pid | grep number

jmap -histo pid|head -100

jstack -l PID >> a.log

jmap -dump:live,format=b,file=/dump.bin pid

• Shallow Size：表示对象自身占用的内存大小，不包括它引用的对象。
• Retained size：当前对象内存大小+当前对象直接或间接引用的对象大小，全部的总和，简单理解，就是当前对象被GC后，总共能释放的内存大小。

本次问题排查

jstack -l PID >> a.log

• RUNNABLE 线程运行中或I/O等待
• BLOCKED 线程在等待monitor锁(synchronized关键字)
• TIMED_WAITING 线程在等待唤醒，但设置了时限
• WAITING 线程在无限等待唤醒

“http-nio-8888-exec-6666” #8833 daemon prio=5 os_prio=0 tid=0x00001f2f0016e100 nid=0x667d waiting on condition [0x00002f1de3c5200]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
– parking to wait for  <0x00000007156a29c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.alibaba.druid.pool.DruidDataSource.takeLast(DruidDataSource.java:1897)
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1458)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1253)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4619)
at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:680)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4615)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1231)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1223)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:90)
at com.baomidou.dynamic.datasource.ds.ItemDataSource.getConnection(ItemDataSource.java:56)
at com.baomidou.dynamic.datasource.ds.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:48)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:82)
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:68)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:84)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
at com.github.pagehelper.PageInterceptor.intercept(PageInterceptor.java:143)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
at com.sun.proxy.$Proxy571.query(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  at com.alibaba.druid.pool.DruidDataSource.takeLast(DruidDataSource.java:1897)
DruidConnectionHolder takeLast() throws InterruptedException, SQLException {
try {
while (poolingCount == 0) {
emptySignal(); // send signal to CreateThread create connection

              if (failFast && isFailContinuous()) {
                  throw new DataSourceNotAvailableException(createError);
              }

              notEmptyWaitThreadCount++;
              if (notEmptyWaitThreadCount > notEmptyWaitThreadPeak) {
                  notEmptyWaitThreadPeak = notEmptyWaitThreadCount;
              }
              try {
                  // 数据库的连接都没有释放且被占用，连接池中无可用连接，导致请求被阻塞
                  notEmpty.await(); // signal by recycle or creator
              } finally {
                  notEmptyWaitThreadCount--;
              }
              notEmptyWaitCount++;

              if (!enable) {
                  connectErrorCountUpdater.incrementAndGet(this);
                  throw new DataSourceDisableException();
              }
          }
      } catch (InterruptedException ie) {
          notEmpty.signal(); // propagate to non-interrupted thread
          notEmptySignalCount++;
          throw ie;
      }

      decrementPoolingCount();
      DruidConnectionHolder last = connections[poolingCount];
      connections[poolingCount] = null;

      return last;
}

try {
  SqlSession sqlSession = sqlSessionFactory.openSession(ExecutorType.BATCH);
  TestMapper mapper = sqlSession.getMapper(TestMapper.class);
  mapper.insetList(list);
  sqlSession.flushStatements();
} catch (Exception e) {
   e.printStackTrace();
}

问题复现

server.tomcat.max-threads=200

server.tomcat.max-connections=10000

server.tomcat.accept-count=100

server.tomcat.min-spare-threads=100

解决

spring: 
  redis:
    host: localhost
    port: 6379
    password: 
  datasource:
    druid:
      stat-view-servlet:
        enabled: true
        loginUsername: admin
        loginPassword: 123456
    dynamic:
      druid:
        initial-size: 5
        min-idle: 5
        maxActive: 20
        maxWait: 60000
        timeBetweenEvictionRunsMillis: 60000
        minEvictableIdleTimeMillis: 300000
        validationQuery: SELECT 1 FROM DUAL
        testWhileIdle: true
        testOnBorrow: false
        testOnReturn: false
        poolPreparedStatements: true
        maxPoolPreparedStatementPerConnectionSize: 20
        filters: stat,slf4j,wall
        connectionProperties: druid.stat.mergeSql\=true;druid.stat.slowSqlMillis\=5000

sqlSession.close();

admin普通

收藏 海报 链接