Why 
It's Important to Recuse Yourself
By following the recusal rules scrupulously and carefully, you will avoid falling into ethical deep water should that "casual lunch" turn into actual negotiations for a position. A negotiation for employment occurs when you and the prospective employer have a mutual interest in a position, regardless of whether you know what the exactly salary will be or the title of the position. When you are negotiating for employment, then the interests of the prospective employer are imputed to you, just as if you owned a lot of stock in them.
You are ethically required to disqualify--or recuse--yourself from participation on official matters that have a financial effect upon the prospective employer. See 5 CFR 2634.604.
If you do not participate in any official matters, then you need not notify anyone. Disqualification is the ethical requirement, not notification. See 5 CFR 2635.604(b).
However, as a prudential matter, please note that the most effective way to ensure that you don't participate in a particular matters is to tell other people so they can help prevent you from having an ethics issue. You could tell your supervisor that you can't work on particular matters that affect the entity or you could issue a recusal statement that tells them and others that you have a conflict.
