My Dad Is Threatening To Take Me To Court

Location: Washington State. When I was 17 my sister had a child, but was not fit to care for the child and my dad and I agreed to take care of him together until we figured out what the next steps were. Fast forward to 2 years later, and my dad legally adopted him. Not long after that he moved to Montana, leaving me at another sisters house with the 2 year old child to raise as my own. He did offer for us to go with him, but I did not want to leave the only place I’ve called home. Fast forward to now, my son is 7 years old, and my dad has come every few months to take him for the weekend. Well it has gotten to a point where I am not comfortable with sending my son with him as he has been demanding when he gets to see him and omitting information when I ask what their plans are. I have expressed my uncomfortability and my dad is threatening to take me to court to work out a parenting plan. He thinks the court would give him more visitation. Does he actually have a case that could go in his favor?

Edit: I don’t receive any child support from him, but I am my sons payee for social security because my dad is on disability. We did have a verbal agreement for him to make me a legal guardian as well, but he later refused to do that. I do have Power of Attorney to take him to the doctor and enroll him in school. I also have all of the legal documents that my dad signed when my son was adopted in my possession. That includes documents stating he would be responsible for taking care and providing for him. I am planning on contacting our local pro bono team in the morning, but unfortunately I cannot afford a lawyer outside of that at this time