Combined naval patrol appeared to be largest such flotilla to approach US territory and ‘highly provocative’, expert says