”...thousands of species, most new to science.". What exactly is 'most'?

In the sentence

That spoon of soil might contain thousands of species, most new to science.

How should one interpret 'most'? Is it "most of which" or "most being"? Or what?

The answer is both: there is a implied partitive fused-head with most being interpreted as most of which - subject in a verbless clause with predicate new to science, comparable in function to a relative clause.

The string most new to science acts as a supplement. A paraphrase with a more canonical clause structure would be

• That spoon of soil might contain thousands of species, and most species in that spoon of soil would be new to science.

Of course, there could be a range of paraphrases here: most of them being..., with most of them being..., of which most would be..., most of which would be... etc. But seeing as the matrix clause has preterite modal might, a paraphrase with a finite clause headed by would captures the full meaning best.

See The Cambridge Grammar of the English Language p1359 for verbless clauses as supplements

5.2 The form of supplements

...

(f) Verbless clause

[28]

i The tourists, most of them foreigners, had been hoarded onto a cattle truck.

...

In [i] the supplement is comparable in function to a relative clause: compare who were most of them foreigners (or most of whom were foreigners). If the supplement consisted of foreigners on its own, it would be an ascriptive NP, like those in [Her father, a die-hard conservative, refused to even consider the proposal / Wilson, Secretary to the Cabinet, had informed the Prime Minister immediately]; most of them, however, does not function as a modifier in NP structure, so most of them foreigners must be analysed as a reduced clause – one which could not stand alone as a sentence.

p411 for implicitly partitive fused-heads

There are three main types of fused-head construction: simple, partitive, and special. The partitive may be explicit, with an of phrase in post-head modifier position, or implicit, with ellipsis of the of phrase. Compare, then:

[5]

i While Kim had lots of books, Pat had [very few]. [simple]

ii [Few of her friends]knew she was ill. [explicitly partitive]

iii We made numerous suggestions but [few]were taken up. [implicitly partitive]

...

Simple and implicitly partitive constructions

These are usually interpreted anaphorically, i.e. via an antecedent. We understand [5i], for example, as “very few books” and [5iii] as “few of the suggestions”.