well first you have to tell us what you already have

sm64

bk/bt

mk64

ge007

kirby64

star fox 64

majoras mask

snowboard kids 2

perfect dark

anything else is free game

oot and mm if for some odd reason you don't have them

okay just oot then

pooper mario
pokemon stadium maybe
f-zero
goemon's adventure
rayman 2

GLOVER

Super Smash Bros
The World is not Enough: 007
Donkey Kong 64
Conker's Bad Fur Day
Pokemon Stadium 1 and 2(though its better to buy them with the extensions and if you have the original games so you aren't stuck with rentals the whole time)
Glover
Battletanx: Global Assualt

Mischief Makers
Doom 64
Buck Bumble
Mischief Makers
Excite Bike 64
either Turok
Mischief Makers
Jet Force Gemini
Mischief Makers
Star Soldier Vanishing Earth
Did I mention Mischief Makers?

DK64
Diddy Kong Racing
Rampage: World Tour

paper mawo

Seconding Glover 64

not enough oot on those lists
glover, pokemon stadium 2, dk64, and ssb would be good too
also oot

Bomberman 64

have obtained the following

f-zero

goemon's adventure

oot

ssb

paper mario

also "legally obtain" really meant "pirate" for those suggesting either pokemon stadium. you know, since the rent pokemon suck.

edit:

and glover

oh come on oot is way more fun on an actual n64
or gamecube if you get the gamecube port

^ This because you get the Master Quest as well. ^

And bonus NES Zelda, if memory serves well.

^ Two separate ports, but yes, you are correct sir. ^