To delete a song, tap Songs on the bottom bar of the music app.
Find the song you want to delete.
Swipe from right to left on that song, and a delete option should appear on the right.
I've never deleted an album, but I assume the process is the same.