Building a generic content-based sports video analysis system remains a challenging problem because of the diversity in sports rules and game features which makes it difficult to discover generic low-level features or high-level modeling algorithms. One possible alternative is to first classify the sports genre and then apply specific sports domain knowledge to perform analysis. In this paper we describe a multi-level framework to automatically recognize the genre of the sports video. The system consists of a Pseudo-2D-HMM classifier using low-level visual/audio features to evaluate the video clips. The experimental results are satisfactory and extension of the framework to a generic sports video analysis system is being implemented.