Google's John Mueller said on Twitter that time to first byte, as a page speed metric, is not used at all by Google in search or search rankings. John said "AFAIK we currently don't use TTFB for anything in search/ranking......
-
Google: We Do Not Use Time To First Byte For Search Rankings
seroundtable.com