Hi Wendy! You can infuse fresh or dried flowers into milk for making soap, just either way, you want to make a very light infusion so you don’t turn your soap brown.
I would definitely infuse dry flowers instead of fresh into oils though, since the moisture from the flowers could cause some bacterial contamination in your oil.
Frangipani flowers don’t grow around here, so I’m not familiar with their properties – I would look them up & double check that they’re not toxic or irritating or poisonous in some way.
If a flower can’t be eaten or used on your skin, then I would avoid putting it in soap or body care products. :)