I was taught that you have to look at who's talking and who they are talking to. In Matthew 19, Jesus was talking to the Jews, and the Law of Moses was given to the Jews and not the Gentiles, even though Matthew is a New Testament book, so the law on marriage and divorce was given the Jews only.
It is also important to know the audience to whom the books were written. Each of the gospels were written after Jesus' death, which had brought about the end of the Old Law (Ephesians 2:13-16; Colossians 2:13-14). Jesus' recorded teachings were not significant to the Jews seeking to remain under the Old Law, but to the Christians longing to understand the New Law. As Mark opened his account, "The beginning of the gospel of Jesus Christ, the Son of God" (Mark 1:1). If Jesus' teachings were exclusive to the Jews, then there would have been no need for the Holy Spirit to remind the apostles of Jesus' words later. "But the Helper, the Holy Spirit, whom the Father will send in My name, He will teach you all things, and bring to your remembrance all that I said to you" (John 14:26). These memories were not recorded to teach the Jews, but to lay the foundations of Christianity.
It is true that Jesus lived among the Jews, for the most part, during his brief stay on the earth. His audience was primarily Jewish people, but it was not exclusively Jewish (e.g. the Samaritan woman (John 4:4ff), the Canaanite woman in Phoenicia (Matthew 15:21ff), the Roman centurion (Matthew 8:5ff), Greeks (John 12:20ff), etc.).
What Jesus taught was the New Law, preparing people for the change that would take place with his death. For example, he explained the change in how worship would be conducted, "But an hour is coming, and now is, when the true worshipers will worship the Father in spirit and truth; for such people the Father seeks to be His worshipers. God is spirit, and those who worship Him must worship in spirit and truth" (John 4:23-24). He taught the disciples how to resolve conflicts in the church (Matthew 18:15-17). He established the apostles' authority. "And Jesus said to them, "Truly I say to you, that you who have followed Me, in the regeneration when the Son of Man will sit on His glorious throne, you also shall sit upon twelve thrones, judging the twelve tribes of Israel"" (Matthew 19:28). He established the need for baptism as an entrance into the church (John 3:5-8) and practiced it with his disciples (John 3:22-4:2).
Then there is the fact that Paul references Jesus' teachings on marriage. "But to the married I give instructions, not I, but the Lord, that the wife should not leave her husband (but if she does leave, she must remain unmarried, or else be reconciled to her husband), and that the husband should not divorce his wife" (I Corinthians 7:10-11). Paul is referring to the teaching that appears in Matthew 19:1-9; Mark 10:1-12; and Luke 16:18. Though in three different gospels, they all concern the same event. Consistency would require that if Mark and Luke's accounts are for the Gentiles, then Matthew's account equally applies. Yes, different details about the same event appear in the different gospels, but that does not mean those differences in details are because some only apply to a limited audience. Paul's reference of this same event shows that the teachings apply to all Christians.
Jesus came to remove the barrier between Jew and Gentile. "For He Himself is our peace, who made both groups into one and broke down the barrier of the dividing wall, by abolishing in His flesh the enmity, which is the Law of commandments contained in ordinances, so that in Himself He might make the two into one new man, thus establishing peace, and might reconcile them both in one body to God through the cross, by it having put to death the enmity" (Ephesians 2:14-16). In Christ there is neither Jew nor Greek (Galatians 3:28). There is no difference in the rules to which Gentiles and Jews must adhere.
What you are really doing is trying to remove one law of Christ that you don't like. "For whoever keeps the whole law and yet stumbles in one point, he has become guilty of all" (James 2:10). It is not for any man to declare that one book in the New Testament does not apply, nor one chapter, nor one teaching.