Venice Beach Boardwalk features street performers, gift shops, restaurants, snack stands, ocean views, Muscle Beach workout area, people-watching, and more. Venice Beach is near Santa Monica so between the two places you could spend a whole day.
From LAX, a bus ride takes 60-90 minutes each way (including transfers). To get to Venice Beach from Santa Monica in around 15 minutes, take Santa Monica Big Blue Bus #1 or # 2.
If you only have time for one attraction, see Santa Monica rather than Venice Beach. If you have all day then visit Venice Beach in the morning or afternoon, and Santa Monica in the afternoon and evening. The reason is that Santa Monica is active after dark, while most Venice Beach stores start closing in the late afternoon.
If you want to stroll around the boardwalk, visit some shops, watch some performers, then walk down to the beach, give yourself around 2-3 hours, plus travel time. So you will realistically need at least a 5 hour layover for this excursion. If you want to see Santa Monica as well, allow an at least additional 3 hours. Don't forget to allow an additional hour to get through security and walking through the terminal at LAX.