HOUSTON, Sept. 2— Six years after Howard R. Hughes died and after his closest relatives agreed to divide his multimillion-dollar estate, two other groups of people named Hughes are trying to persuade a probate court jury that the agreement would deprive them of their just shares.
With the help of a dozen lawyers, a stunning series of stories have been unfolding before the jury of six and an alternate. They seek to disprove the claim of one party to the agreement that would divide the estate.
The party under attack is composed of three sisters in their 50's, who expect to inherit the paternal line's interest under Texas law. The rival claimants' stories seek to prove that the sisters, Barbara Lapp Cameron of Los Angeles, Agnes Lapp Roberts of Cleveland and Elspeth Lapp DePould of Walnut Creek, Calif., are not Howard Hughes's closest surviving relatives on his father's side.
One set of 400 claimants has been gathered by W.A. Jones, 49, proprietor of Heir Finders Service of Nashville. He said that 11 days before Mr, Hughes died April 5, 1976, Jeff Milton Hughes of Houston hired Mr. Jones to draw up a family tree to see how closely he was related to Howard Hughes. Contract to Run a Tree
''We made a contract to run a family tree,'' Mr. Jones said. He talks nonstop, his white teeth sparkling, his blue eyes flashing behind gold framed glasses. ''He gave me $1,000 that day and a contract.''
Then Mr. Hughes died, childless, and a disputed will turned up, and Mr. Jones went on television to challenge its authenticity. ''All the Hugheses across the country began contacting me,'' he said. The interview over lunch in Mr. Jones's hotel a block from the courthouse was constantly interrupted by members of his flock of claimants. He said 70 of them were in the hotel, sleeping sometimes four to a room to save money. Mr. Jones said he had collected $32,421 from them; he said he had spent $1,130,000. ''My deal is 10 per cent and I have to pay attorneys' fees and costs,'' he said.
Although its managers estimate the Hughes estate to be worth $160 million and the Internal Revenue Service estimates it at $460 million, Mr. Jones said he believed it to be worth $2 billion. And if he and his claimants prevail, they will divide half of that, with his end $100 million. He said enemies he did not name have threatened him. Inquiry by Consumer Office
Martha Lehmann, an employe of the Texas Attorney General's Consumer Fraud Division office, said that her office was investigating Mr. Jones's representations to the potential heirs he had lined up.
Mr. Jones's position on how part of the Hughes inheritance is to be won for his backers is straightforward. The Lapp sisters say they are the granddaughters of Rupert Hughes, who was Howard Hughes's uncle. They say that that their mother, Elspeth Hughes Lapp, was Rupert Hughes's only child. One thing that no one disputes is that the three are the daughters of Elspeth Hughes Lapp.
Their lawyers have put into evidence a marriage license between Agnes Hedge and Rupert Hughes, a birth certificate for Elspeth Hughes listing Rupert as father, a decree in which Rupert and Agnes took joint custody of Elsepth when they divorced in New York in 1904. There also were photographs, autographed books and other memorabilia.
There is less here than meets the eye, said Mr. Jones and his counsel, Alex Meacham of Smyrna, Tenn. They assert that Elspeth, who died a suicide in 1946 at the age of 49, was not Rupert's daughter but the child of one of a number of illicit relationships Rupert charged to Agnes in the divorce. 'She Was a Beautiful Thing'
''She was a streetwalker and he proved it,'' Mr. Jones said of Agnes Hughes. ''There was Handsome Jim Beebe in Iowa. She spent three months with him. She was a beautiful thing. I have pictures of her.''
And not only was Agnes Hughes no better than she should be, Mr. Jones said, her husband was physically unable to father children because of mumps in childhood.
The Jones case is now before the court. About 20 witnesses will be called to tell the family stories they have heard about Rupert and Agnes, according to George Parnham, a Houston lawyer who was hired six weeks ago by Mr. Jones because Mr. Meacham is not at home in Texas courts.
Mr. Parnham and his co-counsel, Jacqueline Taylor, won an important ruling from Judge Pat Gregory yesterday that will enable them to present this evidence providing they can qualify their witnesses, one by one, under the strict rules that permit some hearsay.
''If we didn't have that, we might as well quit,'' said Mr. Parnham, who was one of the lawyers who failed to persuade a Nevada jury to accept the disputed will. Now, for a Complicated Tale
The Jones group's claim is clear and simple compared with the story told to the jury Monday and yesterday by the other group seeking to upset the agreement. Robert C. Hughes, 56, a teacher from Columbiana, Ala., tells a tale that begins before the Civil War and includes two switches of persons in the ancestry charts of the Lapp sisters.
Robert Hughes's authority for the crucial elements of his story is a trunk, once owned by his uncle, John Douglas Hughes. The trunk was filled with papers that would show the truth of what he says, he testified, adding that it is a shame it was burned in 1942.
In the early 19th century, Robert Hughes began, two families named Hughes were neighbors in Virginia. They were not related. One was headed by Joshua Hughes and the other by Chesley Hughes. To make it more complex, Robert Hughes said that each family had a son of about the same age. The Chesley Hughes son was named Felix Moner Hughes; the Joshua Hughes son was Felix Turner Hughes.
It has been generally understood that Felix Turner Hughes was Howard Hughes's grandfather. Not so, said Robert Hughes. They Then Switch, He Says
What happened, he said, was that Felix Moner Hughes took to hanging out with the Joshua Hughes family. When the Civil War came, he stayed with them because they were Union supporters while his own family was Confederate, the story goes. Felix Turner Hughes disappeared, perhaps to be killed in the War, and Felix Moner Hughes became known as Felix Turner Hughes and went on to become Howard's grandfather, it is argued.
For the Robert Hughes group, Elspeth Hughes Lapp is still left to be dealt with, because Rupert was Felix Hughes's son and Howard's uncle, no matter who Felix's father was.
Again things are not what they seem, Robert Hughes testified. He said that the real Elspeth drowned in a swimming pool in Los Angeles in 1922. Then, he explained, Rupert's second wife, Eleanor Patterson Dial, substituted her own daughter, who also was named Elspeth, and Rupert went along with it.
Why did Agnes Hughes, then still alive, accept this substitution for her dead daughter? Robert Hughes was asked. ''Probably a financial consideration,'' he testified. A group of 16 of Howard Hughes's relatives from the maternal side have already been designated as the heirs of half his estate. The court action here is over the other half. However, the whole agreement to divide, with its secondary parts, would be inoperative if either challenge succeeds.
Illustrations: Photo of Tom Askins, Charles Ireland and Herbert Askins who claim to be heirs of Howard Hughes