The 95-year-old has dropped her plans of a large gathering in the wake of the Omicron variant of coronavirus.
The Duke and Duchess of Sussex have long felt "sidelined by the institution and were not a fundamental part of its future."
She is yet to meet her great-granddaughter Lilibet Diana, whom Prince Harry and Meghan Markle have named after her.
The British monarch was planning to welcome around 50 relatives including senior members of the Firm for a pre-Christmas lunch at Windsor Castle this week.
The matter keeps the 95-year-old awake at night and she is "understandably very worried" about it.
The British monarch has cancelled the lunch "with regret" as a precautionary measure to save the festive plans of her loved ones.
Early this summer, she started inviting friends to Windsor for lunches and dinners, and would enjoy watching TV with them late into the evening.
The annual lunch, usually held at Buckingham Palace, had to be called off last year due to tough coronavirus restrictions.
The Queen's biographer said compartmentalising also explains the Queen's ability "to be able to still be working at this age."
It is understood that Andrew had "little or no discussion" with the British monarch, or his elder brother Prince Charles.
The Royal Chapel of All Saints stands on the grounds of his stately home Royal Lodge in Windsor Great Park.
The Duke and Duchess of Sussex will be spending Christmas with their children in California.
Author Christopher Andersen claims Prince Harry's memoir poses a dilemma for the royals whether or not to invite him and Meghan Markle to celebrate.
Palace aides plan to have a short gathering on the grounds of Windsor Castle for the queen to share her festive greetings.
The monarch was previously photographed in the exact same spot with her friend Margaret Rhodes.
Leader of the Opposition Sir Keir Starmer said Johnson lacks the "moral authority" to lead the country.
Kate was joined by her husband Prince William and other members of the royal family at the event, but the Queen herself could not join the festivities.
The British monarch has 12 great-grandchildren, the youngest being Princess Beatrice's daughter who was born in September 2021.
She had been working for the monarch's royal household throughout her 69-year reign.
George Osborne said Obama was enjoying himself and having drinks with his friends, which made him nervous about telling him to leave.
It's particularly hard for the Queen to see William and Harry feuding as she took a "close interest in their upbringing" after they lost their mother.
The Duchess of Sussex is lauded for her charitable work and "bravery and boldness in speaking up" about important global issues.