Modeling Heterogeneous Data with Deep Learning