"Traditional" search engine optimization (SEO) is an online marketing strategy that uses different tactics to help websites appear - and rank highly - in search engine results. Google and other search engines use a bunch of different factors to determine if and where websites will appear in the search results.
YouTube, which is owned by Google, operates in a similar way. Its algorithm uses different factors, like keywords and engagement metrics, to determine if and where websites will appear in the search results.
YouTube SEO means that you try different strategies to help your video rank in the YouTube search results.