Regarding Adam and not inheriting his sin: If that were so, wouldn't we all start in the Garden of Eden and each be given a chance to follow the command not to eat of the fruit of the tree of knowledge of good and evil? No. We are sent with Adam. We are all banished from the garden.
In order for what you are saying to be true, the Garden of Eden would have had to be granted to Adam based on merit -- that God owed him the Garden. If you could start with the premise that "God owes all sinless men the Garden of Eden", then you could claim that anyone with the Garden was sinless. You could try to then make the claim that anyone without the Garden was a sinner (technically that wouldn't follow, but just assume for a minute that it does). Only at that point would you be able to claim that since no baby is born into the Garden of Eden, no child must be sinless.
However, Jesus was sinless. Hebrews 4:15, "For we do not have a high priest who is unable to sympathize with our weaknesses, but we have one who has been tempted in every way, just as we are—yet was without sin." And, he was born in Bethlehem of Judea (Matthew 2:1). Since Jesus was sinless and was also not born into the Garden of Eden, it must follow that being born into the Garden of Eden is not the result of being in a sinless state.
To add to the issue we read the curse that Adam was placed under on Genesis 3:17-19, "To Adam he said, "Because you listened to your wife and ate from the tree about which I commanded you, 'You must not eat of it,' "Cursed is the ground because of you; through painful toil you will eat of it all the days of your life. It will produce thorns and thistles for you, and you will eat the plants of the field. By the sweat of your brow you will eat your food until you return to the ground, since from it you were taken; for dust you are and to dust you will return." No where in the curse that God places on Adam for his sin is the concept of banishment from the Garden of Eden. What Adam's sin introduced to all of man was death.
Banishment from the Garden is explained in Genesis 3:22-23, "And the LORD God said, "The man has now become like one of us, knowing good and evil. He must not be allowed to reach out his hand and take also from the tree of life and eat, and live forever." So the LORD God banished him from the Garden of Eden to work the ground from which he had been taken." The banishment from the Garden was only indirectly related to Adam's sin. However, God said that the reason he banished Adam was because he could potentially eat of the tree of life and live.
So, the real connection would be that if there were no sin in the world, there would be no death. Since there is sin in the world, there is death. Paul said while talking about Adam in Romans 5:18, "Therefore, as through one man’s offense judgment came to all men, resulting in condemnation," (NKJV). We all are now subject to judgment because Adam sinned. If Adam had not sinned, then we would not be subject to judgment. Judgment can only go two ways -- guilty or innocent. Since we all have sinned (Romans 3:23), the practical consequence of being subject to judgment is that we all stand condemned.
However, as noted above, Jesus was without sin. He was still subject to judgment because he was of the lineage of Adam (Luke 3). But, because he was not guilty, he was judged to be innocent. Yet -- he died. So, now we are back to the crux of the issue. If Adam's sin was transferred to all men, then how could Jesus be sinless? And if Jesus was sinless, why did he die? Romans 6:23, "For the wages of sin is death." However, if Adam's curse was transferred to all men (rather than Adam's sin), then Jesus dying makes sense. Whether or not we are sinless before God does not change whether or not we will die. Hebrews 9:27, "Just as man is destined to die once, and after that to face judgment."