Abstract
Deep networks have continually outperformed conventional methods on a variety of tasks. However, the unexplainable aspect of deep networks limits its application in the high impact yet high risk tasks, e.g., financial credit scenario, medical diagnosis, terrorism detection and so on. Most existing interpretable approaches focus on heuristic visualization without a clear meaning. To tackle this issue, we propose a general framework to track the prediction making process (or the rules) from the input to the latent representations of deep networks. Given a deep model, the proposed method chooses a group of hidden units by considering both importance and diversity. Each selected unit is then fit by a random forest model to imitate the local structure of the overall model. For a specific input data, all paths from input to the selected units are merged in an intersection and union way to generate the final rules. We experimentally demonstrate that the proposed method can extract rules in various scenarios, ranging from age prediction, scene classification to activity classification, to interpret the prediction process. We further analyze the validity of the rules and visualize the laver-wise rules.