I wouldn't worry about it. This happens to me all the time, as you can see from this picture of the Japanese themed body wash I made with Yuzu fragrance oil. I've had it happen with all kinds of citrus-y fragrance and essential oils, from Sweet Meyer Lemon and Lemon Curd to tangerine or sweet orange. It's just something citrus does with certain ingredients. I haven't had huge issues with lotions or creams, but it definitely alters the colour of things like my surfactant blends all kinds of yellow or orange.
Check out this post on using orange essential oil in your products.
fragrances that contain vanilla, too. They start browning over time, leaving your white lotion beige and your soap a more chocolate-y colour than you expected. This one I made with Michele from Windy Point Soap started off creamy, but the strawberry jam fragrance oil we used turned it brown. Which is fine with me as it smells and feels great!
Having said this, always observe the colour of your product. (This is one of the reasons I like clear bottles.) If you have a product that's changing colour - for instance, going brown or green or pink - that could be unpreserved or has been exposed to a lot of heat, you may be experiencing some kind of contamination.
Product testing - includes information on fragrance morphing
Surfactants and clarity
My article from Handmade magazine, The science behind citrus
My article from Handmade magazine, Understanding the vanilla villain
My article from Handmade magazine, The science of colour morphing
If you're interested in supporting the blog, please check out my Patreon page! There all kinds of rewards for subscribing, from the Q&A section to the requests for duplicating section. $10 subscribes also receive an e-zine every month brimming with new formulas and ingredients!