A regular language L is called dense if the fraction fm of words of length m over some fixed signature that are contained in L tends to 1 as m tends to infinity. We present an algorithm that computes the number of accumulation points of (fm) in polynomial time, if the regular language L is given by a finite deterministic automaton, and can then also efficiently check whether L is dense. Deciding whether the least accumulation point of (fm) is greater than a given rational number, however, is coNP-complete. If the regular language is given by a non-deterministic automaton, checking whether L is dense becomes PSPACE-hard. We will formulate these problems as convergence problems of partially observable Markov chains, and reduce them to combinatorial problems for periodic sequences of rational numbers.